blacksmith - Notice history

All systems operational

100% - uptime

eu-central Storage Cluster - Operational

100% - uptime
Jun 2025 · 100.0%Jul · 100.0%Aug · 100.0%

Previous page

Next page

Jun 2025
Jul 2025
Aug 2025

us-west Storage Cluster - Operational

100% - uptime
Jun 2025 · 100.0%Jul · 100.0%Aug · 100.0%

Previous page

Next page

Jun 2025
Jul 2025
Aug 2025

eu-west Storage Cluster - Operational

100% - uptime
Jun 2025 · 100.0%Jul · 100.0%Aug · 100.0%

Previous page

Next page

Jun 2025
Jul 2025
Aug 2025

API - Operational

100% - uptime
Jun 2025 · 100.0%Jul · 99.98%Aug · 100.0%

Previous page

Next page

Jun 2025
Jul 2025
Aug 2025
100% - uptime

https://blacksmith.sh - Operational

100% - uptime
Jun 2025 · 100.0%Jul · 100.0%Aug · 100.0%

Previous page

Next page

Jun 2025
Jul 2025
Aug 2025

Github → Actions - Operational

Github → API Requests - Operational

Github → Webhooks - Operational

Codesmith - Operational

100% - uptime
Jun 2025 · 100.0%Jul · 100.0%Aug · 100.0%

Previous page

Next page

Jun 2025
Jul 2025
Aug 2025

Dashboard - Operational

100% - uptime
Jun 2025 · 100.0%Jul · 100.0%Aug · 100.0%

Previous page

Next page

Jun 2025
Jul 2025
Aug 2025

Notice history

Aug 2025

Timeouts reported when downloading certain actions from GitHub
  • Resolved
    UTC
    Resolved

    This incident has been resolved, we are no longer seeing signs of this flake with GitHub.

  • Investigating
    UTC
    Investigating

    We are being alerted of timeouts when downloading the different actions used in your workflow file from GitHub. This usually indicates some sort of upstream degradation in GitHub's control plane

Degradation in US West storage cluster
  • Resolved
    UTC
    Resolved
    This incident has been resolved.
  • Monitoring
    UTC
    Monitoring

    Fix has been deployed, and we are currently monitoring the state of the cluster.

  • Identified
    UTC
    Identified

    We're about to deploy a fix, this will result in a few minutes of downtime and all stickydisk artifacts being cleared.

  • Monitoring
    UTC
    Monitoring

    We are still seeing signs of degradation in bursts as the storage cluster trends towards recovery. We will continue monitoring the situation and provide updates here.

  • Resolved
    UTC
    Resolved

    This incident has been resolved. We had a sudden surge of usage that required us to scale up our storage cluster.

  • Investigating
    UTC
    Investigating
    We are currently investigating this incident.
Seeing an elevated rates of failures for Docker builds in our EU region
  • Resolved
    UTC
    Resolved
    This incident has been resolved.
  • Investigating
    UTC
    Investigating
    We are currently investigating this incident.
Seeing elevated errors in Blacksmith Docker build cache setup
  • Resolved
    UTC
    Resolved

    This incident has been resolved Docker builds should now be working as expected.

  • Investigating
    UTC
    Investigating
    We are currently investigating this incident.
Delays with job adoption
  • Postmortem
    UTC
    Postmortem

    What happened: A bug in our job reconciliation system (which ensures we spin up VMs to run your CI jobs even if GitHub fails to send a webhook) caused us to overprovision VMs, spinning up 110% more VMs than needed. This additional load led to increased queue times and made it harder for the system to recover quickly due to delays in draining the additional VMs.

    What we’ve done since:

    • Fixed the underlying bug

    • Improved how we drain the queue under load

    • Added safeguards to better handle similar failure modes in the future

  • Resolved
    UTC
    Resolved

    This incident has been resolved, queue times are returning to normal.

  • Identified
    UTC
    Identified

    We are running into fleet capacity issues in our amd64 cluster, and are looking into mitigations.

  • Investigating
    UTC
    Investigating

    We're seeing delays in job adoption times for AMD64 jobs across both our EU and U.S. regions. We're currently investigating this issue.

Jul 2025

GitHub returning 429s when downloading first party actions
  • Update
    UTC
    Update
    This incident has been resolved.
  • Resolved
    UTC
    Resolved

    This incident has been resolved by GitHub.

  • Identified
    UTC
    Identified
    GitHub have declared an upstream incident, we will continue monitoring these failures we are seeing sporadically.
  • Investigating
    UTC
    Investigating

    We are monitoring reports of GitHub returning 429s when downloading first party actions such as actions/setup-go etc. The error string being reported is of the form:
    ```
    Error: Response status code does not indicate success: 429 (Too Many Requests). DC04:39E04E:636AE0:858F56:6887975E
    ```

Increased queueing in our US region
  • Resolved
    UTC
    Resolved

    This incident has been resolved, queue times are back to normal.

  • Investigating
    UTC
    Investigating

    We are currently investigating signs of increased queue times in our US region.

Degradation in control plane may cause dashboard to load slower
  • Resolved
    UTC
    Resolved

    This incident has been resolved, we've deployed a mitigation that has stabilized our control plane.

  • Update
    UTC
    Update

    Users may also see degraded performance interacting with our cache endpoints, we are investigating the incident.

  • Investigating
    UTC
    Investigating
    We are currently investigating this incident.
Control plane outage preventing job adoption
  • Resolved
    UTC
    Resolved
    This incident has been resolved.
  • Monitoring
    UTC
    Monitoring
    We implemented a fix and are currently monitoring the result.
  • Investigating
    UTC
    Investigating

    We have been notified about an outage in our control plane that is preventing jobs from being assigned to our runners. We are working on a fix to restore service imminently.

Occasional failures during setup of GitHub Actions jobs on Blacksmith runners
  • Resolved
    UTC
    Resolved

    This incident has been resolved we will be gradually re-enabling transparent caching for all orgs.

  • Monitoring
    UTC
    Monitoring

    We are implementing a fix to re-enable transparent caching for all orgs. There should be no customer facing impact any more.

  • Identified
    UTC
    Identified

    The issue appears to be related to the setup of our caching backend. We will temporarily disable transparent caching to prevent setup of GitHub Actions from failing. This will not affect customers that are using our useblacksmith/cache or useblacksmith/setup-* actions.

  • Investigating
    UTC
    Investigating

    We are being notified of occasional failures in the setup step of GitHub Actions jobs on Blacksmith runners. We are investigating the cause of this flakiness.

Jun 2025

Landing page undergoing maintenance
  • Completed
    June 30, 2025 at 10:16 PMUTC
    Completed
    June 30, 2025 at 10:16 PMUTC
    Maintenance has completed successfully.
  • In progress
    June 30, 2025 at 9:30 PMUTC
    In progress
    June 30, 2025 at 9:30 PMUTC

    There is ongoing maintenance on our landing page (blacksmith.sh), but our dashboard and support portals are unaffected https://app.blacksmith.sh/

Seeing timeouts in downloading some third-party actions in GitHub Actions
  • Resolved
    UTC
    Resolved

    This incident has been resolved, our customers are no longer seeing these slowdowns or errors when downloading third party actions.

  • Investigating
    UTC
    Investigating

    Customers are reporting timeouts in downloading some actions during job runs. This implies an issue upstream with GitHub, we are continuing to monitor the situation.

Potential GitHub outage preventing jobs from being assigned to our runners
  • Resolved
    UTC
    Resolved

    GitHub have marked their incident as closed, our runners are back to behaving as expected.

  • Update
    UTC
    Update

    GitHub has declared a widespread incident that is affecting Actions as well, we will continue to monitor their status page for updates - https://www.githubstatus.com/incidents/y7lb2rg4btd7

  • Investigating
    UTC
    Investigating

    We are seeing an uptick in upstream errors from the GitHub control plane, preventing runners from being assigned jobs. We are monitoring the situation.

Increased queue times for runners
  • Resolved
    UTC
    Resolved
    This incident has been resolved.
  • Investigating
    UTC
    Investigating

    We are noticing increased queue times for runners in our EU region. We are currently investigating this incident.

Degraded performance of our incremental docker builds in our EU region
  • Resolved
    UTC
    Resolved

    This incident has been resolved, the fix has been deployed to our storage cluster and builds in the EU region are working as expected.

  • Monitoring
    UTC
    Monitoring

    We have deployed a fix and are monitoring the cluster as it comes back up.

  • Investigating
    UTC
    Investigating
    We are currently investigating this incident.

Jun 2025 to Aug 2025

Next