Status | Blacksmith - Notice history

All systems operational

Blacksmith Managed Runners - Operational

100% - uptime
Jul 2025 · 99.98%Aug · 99.98%Sep · 99.89%

Previous page

Next page

Jul 2025
Aug 2025
Sep 2025

Incremental Docker Builders - Operational

100% - uptime
Jul 2025 · 99.98%Aug · 99.99%Sep · 99.89%

Previous page

Next page

Jul 2025
Aug 2025
Sep 2025

API - Operational

100% - uptime
Jul 2025 · 99.98%Aug · 100.0%Sep · 99.89%

Previous page

Next page

Jul 2025
Aug 2025
Sep 2025

Website - Operational

100% - uptime
Jul 2025 · 100.0%Aug · 100.0%Sep · 99.98%

Previous page

Next page

Jul 2025
Aug 2025
Sep 2025

Github → Actions - Operational

Github → API Requests - Operational

Github → Webhooks - Operational

Notice history

Sep 2025

Blacksmith control plane is unresponsive.
  • Resolved
    Resolved

    This incident has been resolved, queue times have returned to normal.

  • Monitoring
    Monitoring

    We are seeing some recovery and the system is working through the backlog of the queue. You may continue to see high queue times until the backlog is cleared. We are actively monitoring the situation.

  • Update
    Update

    We're continuing to investigate the slowness to some of our control plane endpoints. Users may continue to notice longer than normal queue times. We are actively investigating what is causing this degradation to alleviate the effect on customer CI jobs.

  • Identified
    Identified
    Interactions with our control plane remain degraded. We are seeing customer jobs getting picked up but users might see longer than normal queue times and longer load times on our dashboard.
  • Investigating
    Investigating
    We are currently investigating this incident.
Cache degradation in the EU cluster
  • Resolved
    Resolved

    This incident has been resolved, cache operations are back to normal in the EU.

  • Monitoring
    Monitoring
    We have identified an issue with our EU storage cluster and have deployed a fix. We are seeing recovery and will continue monitoring performance.
  • Investigating
    Investigating

    We're seeing degradation in our colocated cache in the EU. We are currently investigating this incident.

GitHub Actions is having an outage
  • Resolved
    Resolved

    This incident has been resolved, queue times are back to normal.

  • Monitoring
    Monitoring

    GitHub has resolved the incident, and job dispatch has returned to normal. Our infrastructure is fully operational and new jobs are being picked up without delay. Earlier, you may have noticed some queued jobs being canceled as part of our mitigation efforts to prevent long backlogs -- this ensured new jobs could start processing quickly. Retrying canceled jobs should see them run normally once again. We’ll continue to keep an eye on metrics, but no further customer impact is expected. We apologize for the inconvenience.

  • Identified
    Identified

    We are about to issue a cancelation for some subset of queued jobs across orgs. This is in an effort to bring queue times back to normal by trimming down the backlog. Customers can retry the canceled jobs and expect normal operation thereafter.

  • Update
    Update
    Even though GitHub has reported their incident as resolved, but we’re still seeing delayed job starts for a fraction of jobs. Things are steadily improving as the backlog drains, but you may continue to see slower job starts until recovery is complete.
  • Update
    Update

    We're seeing a massive backlog of queued jobs due to the outage, which is now slowly draining, so you may see jobs being slower to get picked up for the next few minutes.

  • Monitoring
    Monitoring

    GitHub has declared an incident of their own for this, and we're seeing some percentage of jobs not being picked up at all. We're monitoring the impact of this incident on our customers.

GitHub Actions checkout are taking longer than usual
  • Resolved
    Resolved

    This incident has been resolved. We will be providing a postmortem to affected customers by tomorrow.

  • Monitoring
    Monitoring
    We implemented a fix and are currently monitoring the result.
  • Identified
    Identified
    We are continuing to work on a fix for this incident.
  • Monitoring
    Monitoring

    Our upstream ISP identified some issues between them and their network vendor. They have made changes to mitigate these blips while working on a longer term resolution. We will share more details as we receive them. All network operations in Blacksmith runners have returned to normal. We are actively monitoring the situation.

  • Update
    Update

    We are being alerted about intermittent degradation in our network stack in one of our regions. This may be sporadically manifesting as slower docker pulls, interactions with certain language mirrors, git checkouts. We are investigating the source of this issue.

  • Investigating
    Investigating
    We're hearing this from several customers today that they are seeing elevated `actions/checkout` times. We are noticing this in our local development as well. This is very likely an upstream degradation with the GItHub control plane but we are investigating what we can do to mitigate these.
Elevated queue times for jobs in the EU region
  • Resolved
    Resolved
    This incident has been resolved.
  • Investigating
    Investigating
    We are currently investigating this incident.

Aug 2025

Timeouts reported when downloading certain actions from GitHub
  • Resolved
    Resolved

    This incident has been resolved, we are no longer seeing signs of this flake with GitHub.

  • Investigating
    Investigating

    We are being alerted of timeouts when downloading the different actions used in your workflow file from GitHub. This usually indicates some sort of upstream degradation in GitHub's control plane

Degradation in US West storage cluster
  • Resolved
    Resolved
    This incident has been resolved.
  • Monitoring
    Monitoring

    Fix has been deployed, and we are currently monitoring the state of the cluster.

  • Identified
    Identified

    We're about to deploy a fix, this will result in a few minutes of downtime and all stickydisk artifacts being cleared.

  • Monitoring
    Monitoring

    We are still seeing signs of degradation in bursts as the storage cluster trends towards recovery. We will continue monitoring the situation and provide updates here.

  • Resolved
    Resolved

    This incident has been resolved. We had a sudden surge of usage that required us to scale up our storage cluster.

  • Investigating
    Investigating
    We are currently investigating this incident.
Seeing an elevated rates of failures for Docker builds in our EU region
  • Resolved
    Resolved
    This incident has been resolved.
  • Investigating
    Investigating
    We are currently investigating this incident.
Seeing elevated errors in Blacksmith Docker build cache setup
  • Resolved
    Resolved

    This incident has been resolved Docker builds should now be working as expected.

  • Investigating
    Investigating
    We are currently investigating this incident.
Delays with job adoption
  • Postmortem
    Postmortem

    What happened: A bug in our job reconciliation system (which ensures we spin up VMs to run your CI jobs even if GitHub fails to send a webhook) caused us to overprovision VMs, spinning up 110% more VMs than needed. This additional load led to increased queue times and made it harder for the system to recover quickly due to delays in draining the additional VMs.

    What we’ve done since:

    • Fixed the underlying bug

    • Improved how we drain the queue under load

    • Added safeguards to better handle similar failure modes in the future

  • Resolved
    Resolved

    This incident has been resolved, queue times are returning to normal.

  • Identified
    Identified

    We are running into fleet capacity issues in our amd64 cluster, and are looking into mitigations.

  • Investigating
    Investigating

    We're seeing delays in job adoption times for AMD64 jobs across both our EU and U.S. regions. We're currently investigating this issue.

Jul 2025

GitHub returning 429s when downloading first party actions
  • Update
    Update
    This incident has been resolved.
  • Resolved
    Resolved

    This incident has been resolved by GitHub.

  • Identified
    Identified
    GitHub have declared an upstream incident, we will continue monitoring these failures we are seeing sporadically.
  • Investigating
    Investigating

    We are monitoring reports of GitHub returning 429s when downloading first party actions such as actions/setup-go etc. The error string being reported is of the form:
    ```
    Error: Response status code does not indicate success: 429 (Too Many Requests). DC04:39E04E:636AE0:858F56:6887975E
    ```

Increased queueing in our US region
  • Resolved
    Resolved

    This incident has been resolved, queue times are back to normal.

  • Investigating
    Investigating

    We are currently investigating signs of increased queue times in our US region.

Degradation in control plane may cause dashboard to load slower
  • Resolved
    Resolved

    This incident has been resolved, we've deployed a mitigation that has stabilized our control plane.

  • Update
    Update

    Users may also see degraded performance interacting with our cache endpoints, we are investigating the incident.

  • Investigating
    Investigating
    We are currently investigating this incident.
Control plane outage preventing job adoption
  • Resolved
    Resolved
    This incident has been resolved.
  • Monitoring
    Monitoring
    We implemented a fix and are currently monitoring the result.
  • Investigating
    Investigating

    We have been notified about an outage in our control plane that is preventing jobs from being assigned to our runners. We are working on a fix to restore service imminently.

Occasional failures during setup of GitHub Actions jobs on Blacksmith runners
  • Resolved
    Resolved

    This incident has been resolved we will be gradually re-enabling transparent caching for all orgs.

  • Monitoring
    Monitoring

    We are implementing a fix to re-enable transparent caching for all orgs. There should be no customer facing impact any more.

  • Identified
    Identified

    The issue appears to be related to the setup of our caching backend. We will temporarily disable transparent caching to prevent setup of GitHub Actions from failing. This will not affect customers that are using our useblacksmith/cache or useblacksmith/setup-* actions.

  • Investigating
    Investigating

    We are being notified of occasional failures in the setup step of GitHub Actions jobs on Blacksmith runners. We are investigating the cause of this flakiness.

Jul 2025 to Sep 2025

Next