Status | Blacksmith - Notice history

All systems operational

Blacksmith Managed Runners - Operational

100% - uptime
Jul 1900 · 100.0%Aug · 100.0%Sep · 100.0%

Previous page

Next page

Jul 1900
Aug 1900
Sep 1900

Incremental Docker Builders - Operational

100% - uptime
Jul 1900 · 100.0%Aug · 100.0%Sep · 100.0%

Previous page

Next page

Jul 1900
Aug 1900
Sep 1900

API - Operational

100% - uptime
Jul 1900 · 100.0%Aug · 100.0%Sep · 100.0%

Previous page

Next page

Jul 1900
Aug 1900
Sep 1900

Website - Operational

100% - uptime
Jul 1900 · 100.0%Aug · 100.0%Sep · 100.0%

Previous page

Next page

Jul 1900
Aug 1900
Sep 1900

Github → Actions - Operational

Github → API Requests - Operational

Github → Webhooks - Operational

Notice history

Sep 2025

GitHub reporting upstream degradation in Actions
  • Resolved
    Resolved

    This incident has been resolved, queue times have returned to normal.

  • Identified
    Identified

    GitHub has resolved the incident but we're seeing some downstream backlog of jobs being assigned to our runners. This is causing increased queue times for some orgs running in the EU.

  • Investigating
    Investigating

    GitHub has declared an incident with their Actions service. We are monitoring for any downstream impact on Blacksmith runners - https://www.githubstatus.com/incidents/s9t1dgymd4v9

Degradation in queue times causing runners to sporadically not get assigned a job
  • Resolved
    Resolved

    This incident has been resolved, we have requested our database provider for a postmortem of the upstream incident.

  • Monitoring
    Monitoring

    Our database provider has rolled out a fix and is claiming recovery. We are also seeing signs of recovery on our end.

  • Identified
    Identified

    Our database provider has declared an incident that is directly affecting our control plane - https://status.supabase.com/incidents/kfrmjq5z9n8x. We are continuing to work with them on a timeline for a fix.

  • Update
    Update
    We are continuing to investigate the cause of this incident. All signs point to a degradation in our upstream database provider. We are actively working with them to resolve the incident.
  • Investigating
    Investigating

    The deployed fix has not completely mitigated the outage. Customers may also be noticing degraded load times for app.blacksmith.sh

  • Identified
    Identified

    We have identified the issue as likely being our database connection pool limits, we are rolling out a fix to see if it helps with the situation.

  • Investigating
    Investigating
    We are currently investigating this incident.

Aug 2025

Timeouts reported when downloading certain actions from GitHub
  • Resolved
    Resolved

    This incident has been resolved, we are no longer seeing signs of this flake with GitHub.

  • Investigating
    Investigating

    We are being alerted of timeouts when downloading the different actions used in your workflow file from GitHub. This usually indicates some sort of upstream degradation in GitHub's control plane

Degradation in US West storage cluster
  • Resolved
    Resolved
    This incident has been resolved.
  • Monitoring
    Monitoring

    Fix has been deployed, and we are currently monitoring the state of the cluster.

  • Identified
    Identified

    We're about to deploy a fix, this will result in a few minutes of downtime and all stickydisk artifacts being cleared.

  • Monitoring
    Monitoring

    We are still seeing signs of degradation in bursts as the storage cluster trends towards recovery. We will continue monitoring the situation and provide updates here.

  • Resolved
    Resolved

    This incident has been resolved. We had a sudden surge of usage that required us to scale up our storage cluster.

  • Investigating
    Investigating
    We are currently investigating this incident.
Seeing an elevated rates of failures for Docker builds in our EU region
  • Resolved
    Resolved
    This incident has been resolved.
  • Investigating
    Investigating
    We are currently investigating this incident.
Seeing elevated errors in Blacksmith Docker build cache setup
  • Resolved
    Resolved

    This incident has been resolved Docker builds should now be working as expected.

  • Investigating
    Investigating
    We are currently investigating this incident.
Delays with job adoption
  • Postmortem
    Postmortem

    What happened: A bug in our job reconciliation system (which ensures we spin up VMs to run your CI jobs even if GitHub fails to send a webhook) caused us to overprovision VMs, spinning up 110% more VMs than needed. This additional load led to increased queue times and made it harder for the system to recover quickly due to delays in draining the additional VMs.

    What we’ve done since:

    • Fixed the underlying bug

    • Improved how we drain the queue under load

    • Added safeguards to better handle similar failure modes in the future

  • Resolved
    Resolved

    This incident has been resolved, queue times are returning to normal.

  • Identified
    Identified

    We are running into fleet capacity issues in our amd64 cluster, and are looking into mitigations.

  • Investigating
    Investigating

    We're seeing delays in job adoption times for AMD64 jobs across both our EU and U.S. regions. We're currently investigating this issue.

Jul 2025

GitHub returning 429s when downloading first party actions
  • Update
    Update
    This incident has been resolved.
  • Resolved
    Resolved

    This incident has been resolved by GitHub.

  • Identified
    Identified
    GitHub have declared an upstream incident, we will continue monitoring these failures we are seeing sporadically.
  • Investigating
    Investigating

    We are monitoring reports of GitHub returning 429s when downloading first party actions such as actions/setup-go etc. The error string being reported is of the form:
    ```
    Error: Response status code does not indicate success: 429 (Too Many Requests). DC04:39E04E:636AE0:858F56:6887975E
    ```

Increased queueing in our US region
  • Resolved
    Resolved

    This incident has been resolved, queue times are back to normal.

  • Investigating
    Investigating

    We are currently investigating signs of increased queue times in our US region.

Degradation in control plane may cause dashboard to load slower
  • Resolved
    Resolved

    This incident has been resolved, we've deployed a mitigation that has stabilized our control plane.

  • Update
    Update

    Users may also see degraded performance interacting with our cache endpoints, we are investigating the incident.

  • Investigating
    Investigating
    We are currently investigating this incident.
Control plane outage preventing job adoption
  • Resolved
    Resolved
    This incident has been resolved.
  • Monitoring
    Monitoring
    We implemented a fix and are currently monitoring the result.
  • Investigating
    Investigating

    We have been notified about an outage in our control plane that is preventing jobs from being assigned to our runners. We are working on a fix to restore service imminently.

Occasional failures during setup of GitHub Actions jobs on Blacksmith runners
  • Resolved
    Resolved

    This incident has been resolved we will be gradually re-enabling transparent caching for all orgs.

  • Monitoring
    Monitoring

    We are implementing a fix to re-enable transparent caching for all orgs. There should be no customer facing impact any more.

  • Identified
    Identified

    The issue appears to be related to the setup of our caching backend. We will temporarily disable transparent caching to prevent setup of GitHub Actions from failing. This will not affect customers that are using our useblacksmith/cache or useblacksmith/setup-* actions.

  • Investigating
    Investigating

    We are being notified of occasional failures in the setup step of GitHub Actions jobs on Blacksmith runners. We are investigating the cause of this flakiness.

Jul 2025 to Sep 2025

Next