Status | Blacksmith - Notice history

All systems operational

Blacksmith Managed Runners - Operational

100% - uptime
Aug 2025 · 99.98%Sep · 99.89%Oct · 100.0%

Previous page

Next page

Aug 2025
Sep 2025
Oct 2025

Incremental Docker Builders - Operational

100% - uptime
Aug 2025 · 99.99%Sep · 99.89%Oct · 99.99%

Previous page

Next page

Aug 2025
Sep 2025
Oct 2025

API - Operational

100% - uptime
Aug 2025 · 100.0%Sep · 99.89%Oct · 100.0%

Previous page

Next page

Aug 2025
Sep 2025
Oct 2025

Website - Operational

100% - uptime
Aug 2025 · 100.0%Sep · 99.98%Oct · 100.0%

Previous page

Next page

Aug 2025
Sep 2025
Oct 2025

Github → Actions - Operational

Github → API Requests - Operational

Github → Webhooks - Operational

Notice history

Oct 2025

Degraded Cache Interactions in the EU
  • Resolved
    Resolved
    This incident has been resolved.
  • Identified
    Identified

    We've identified the issue and are working on a fix.

  • Investigating
    Investigating
    We are currently investigating this incident.
Docker Builds are experiencing Degraded Performance
  • Resolved
    Resolved
    This incident has been resolved.
  • Monitoring
    Monitoring

    We're seeing recovery in our storage cluster. As part of this recovery, we had to clear the cluster and sticky disks have been cleared. Docker Builds should now work as expected. We're continuing to monitor.

  • Update
    Update

    We're continuing to experience issues with our storage cluster which is effecting queue times. We are working towards resolution.

  • Identified
    Identified

    We're beginning to see recovery in our storage cluster. We're continuing to monitor.

  • Investigating
    Investigating

    We are currently investigating this incident and scaling out our storage cluster.

Github Actions are experiencing Degraded Performance
  • Resolved
    Resolved
    This incident has been resolved.
  • Update
    Update

    Github has reported they are seeing full recovery in many systems, but delays are still expected for actions. We will continue to update.

  • Update
    Update

    Github has identified a faulty network component as the cause and is seeing recovery of some services. API Requests are now operating normally.

  • Update
    Update

    GitHub is experiencing major outages across a variety of services (including Webhooks, Actions, and API Requests). This outage is impacting our ability to process jobs since we rely on Webhooks. We are continuing to monitor the situation closely and will provide updates as we receive them.

  • Update
    Update

    We're seeing major performance degradation due to an outage from Github Actions and Webhooks. We are continuing to monitor for updates.

  • Monitoring
    Monitoring

    Github Actions is experiencing degraded performance. We're monitoring Github's incident here: https://www.githubstatus.com/incidents/k7bhmjkblcwp

Longer than normal queue times in our US region
  • Resolved
    Resolved

    This incident has been resolved and queue times have returned to normal.

  • Update
    Update

    Queue times continue to improve. We are continuing to monitor runner queues.

  • Monitoring
    Monitoring

    Queue times are improving. We are continuing to monitor the situation closely.

  • Update
    Update

    We've identified the issue and are working on rolling out a fix shortly.

  • Identified
    Identified

    We've identified a regression and are in the process of rolling out a fix.

  • Investigating
    Investigating

    We are getting notified about longer than normal queue times in our US region. We are investigating the degradation in queue times.

Docker build caching in a degraded state
  • Resolved
    Resolved
    This incident has been resolved.
  • Monitoring
    Monitoring
    We implemented a fix and are currently monitoring the result.
  • Investigating
    Investigating
    We are currently investigating this incident.

Sep 2025

Blacksmith control plane is unresponsive.
  • Resolved
    Resolved

    This incident has been resolved, queue times have returned to normal.

  • Monitoring
    Monitoring

    We are seeing some recovery and the system is working through the backlog of the queue. You may continue to see high queue times until the backlog is cleared. We are actively monitoring the situation.

  • Update
    Update

    We're continuing to investigate the slowness to some of our control plane endpoints. Users may continue to notice longer than normal queue times. We are actively investigating what is causing this degradation to alleviate the effect on customer CI jobs.

  • Identified
    Identified
    Interactions with our control plane remain degraded. We are seeing customer jobs getting picked up but users might see longer than normal queue times and longer load times on our dashboard.
  • Investigating
    Investigating
    We are currently investigating this incident.
Cache degradation in the EU cluster
  • Resolved
    Resolved

    This incident has been resolved, cache operations are back to normal in the EU.

  • Monitoring
    Monitoring
    We have identified an issue with our EU storage cluster and have deployed a fix. We are seeing recovery and will continue monitoring performance.
  • Investigating
    Investigating

    We're seeing degradation in our colocated cache in the EU. We are currently investigating this incident.

GitHub Actions is having an outage
  • Resolved
    Resolved

    This incident has been resolved, queue times are back to normal.

  • Monitoring
    Monitoring

    GitHub has resolved the incident, and job dispatch has returned to normal. Our infrastructure is fully operational and new jobs are being picked up without delay. Earlier, you may have noticed some queued jobs being canceled as part of our mitigation efforts to prevent long backlogs -- this ensured new jobs could start processing quickly. Retrying canceled jobs should see them run normally once again. We’ll continue to keep an eye on metrics, but no further customer impact is expected. We apologize for the inconvenience.

  • Identified
    Identified

    We are about to issue a cancelation for some subset of queued jobs across orgs. This is in an effort to bring queue times back to normal by trimming down the backlog. Customers can retry the canceled jobs and expect normal operation thereafter.

  • Update
    Update
    Even though GitHub has reported their incident as resolved, but we’re still seeing delayed job starts for a fraction of jobs. Things are steadily improving as the backlog drains, but you may continue to see slower job starts until recovery is complete.
  • Update
    Update

    We're seeing a massive backlog of queued jobs due to the outage, which is now slowly draining, so you may see jobs being slower to get picked up for the next few minutes.

  • Monitoring
    Monitoring

    GitHub has declared an incident of their own for this, and we're seeing some percentage of jobs not being picked up at all. We're monitoring the impact of this incident on our customers.

GitHub Actions checkout are taking longer than usual
  • Resolved
    Resolved

    This incident has been resolved. We will be providing a postmortem to affected customers by tomorrow.

  • Monitoring
    Monitoring
    We implemented a fix and are currently monitoring the result.
  • Identified
    Identified
    We are continuing to work on a fix for this incident.
  • Monitoring
    Monitoring

    Our upstream ISP identified some issues between them and their network vendor. They have made changes to mitigate these blips while working on a longer term resolution. We will share more details as we receive them. All network operations in Blacksmith runners have returned to normal. We are actively monitoring the situation.

  • Update
    Update

    We are being alerted about intermittent degradation in our network stack in one of our regions. This may be sporadically manifesting as slower docker pulls, interactions with certain language mirrors, git checkouts. We are investigating the source of this issue.

  • Investigating
    Investigating
    We're hearing this from several customers today that they are seeing elevated `actions/checkout` times. We are noticing this in our local development as well. This is very likely an upstream degradation with the GItHub control plane but we are investigating what we can do to mitigate these.
Elevated queue times for jobs in the EU region
  • Resolved
    Resolved
    This incident has been resolved.
  • Investigating
    Investigating
    We are currently investigating this incident.

Aug 2025

Timeouts reported when downloading certain actions from GitHub
  • Resolved
    Resolved

    This incident has been resolved, we are no longer seeing signs of this flake with GitHub.

  • Investigating
    Investigating

    We are being alerted of timeouts when downloading the different actions used in your workflow file from GitHub. This usually indicates some sort of upstream degradation in GitHub's control plane

Degradation in US West storage cluster
  • Resolved
    Resolved
    This incident has been resolved.
  • Monitoring
    Monitoring

    Fix has been deployed, and we are currently monitoring the state of the cluster.

  • Identified
    Identified

    We're about to deploy a fix, this will result in a few minutes of downtime and all stickydisk artifacts being cleared.

  • Monitoring
    Monitoring

    We are still seeing signs of degradation in bursts as the storage cluster trends towards recovery. We will continue monitoring the situation and provide updates here.

  • Resolved
    Resolved

    This incident has been resolved. We had a sudden surge of usage that required us to scale up our storage cluster.

  • Investigating
    Investigating
    We are currently investigating this incident.
Seeing an elevated rates of failures for Docker builds in our EU region
  • Resolved
    Resolved
    This incident has been resolved.
  • Investigating
    Investigating
    We are currently investigating this incident.
Seeing elevated errors in Blacksmith Docker build cache setup
  • Resolved
    Resolved

    This incident has been resolved Docker builds should now be working as expected.

  • Investigating
    Investigating
    We are currently investigating this incident.
Delays with job adoption
  • Postmortem
    Postmortem

    What happened: A bug in our job reconciliation system (which ensures we spin up VMs to run your CI jobs even if GitHub fails to send a webhook) caused us to overprovision VMs, spinning up 110% more VMs than needed. This additional load led to increased queue times and made it harder for the system to recover quickly due to delays in draining the additional VMs.

    What we’ve done since:

    • Fixed the underlying bug

    • Improved how we drain the queue under load

    • Added safeguards to better handle similar failure modes in the future

  • Resolved
    Resolved

    This incident has been resolved, queue times are returning to normal.

  • Identified
    Identified

    We are running into fleet capacity issues in our amd64 cluster, and are looking into mitigations.

  • Investigating
    Investigating

    We're seeing delays in job adoption times for AMD64 jobs across both our EU and U.S. regions. We're currently investigating this issue.

Aug 2025 to Oct 2025

Next