blacksmith - Notice history

All systems operational

100% - uptime

eu-central Storage Cluster - Operational

100% - uptime
Apr 2026 · 100.0%May · 100.0%Jun · 100.0%

Previous page

Next page

Apr 2026
May 2026
Jun 2026

us-west Storage Cluster - Operational

100% - uptime
Apr 2026 · 100.0%May · 100.0%Jun · 100.0%

Previous page

Next page

Apr 2026
May 2026
Jun 2026

eu-west Storage Cluster - Operational

100% - uptime
Apr 2026 · 100.0%May · 100.0%Jun · 100.0%

Previous page

Next page

Apr 2026
May 2026
Jun 2026

API - Operational

100% - uptime
Apr 2026 · 100.0%May · 100.0%Jun · 100.0%

Previous page

Next page

Apr 2026
May 2026
Jun 2026
100% - uptime

https://blacksmith.sh - Operational

100% - uptime
Apr 2026 · 100.0%May · 100.0%Jun · 100.0%

Previous page

Next page

Apr 2026
May 2026
Jun 2026

Github → Actions - Operational

Github → API Requests - Operational

Github → Webhooks - Operational

Codesmith - Operational

100% - uptime
Apr 2026 · 100.0%May · 100.0%Jun · 100.0%

Previous page

Next page

Apr 2026
May 2026
Jun 2026

Dashboard - Operational

100% - uptime
Apr 2026 · 100.0%May · 100.0%Jun · 100.0%

Previous page

Next page

Apr 2026
May 2026
Jun 2026

Notice history

Jun 2026

Runners hitting Github rate limiting
  • Resolved
    UTC
    Resolved
    This incident has been resolved.
  • Monitoring
    UTC
    Monitoring

    Queueing improving and currently monitoring.

  • Identified
    UTC
    Identified

    We are have mitigated the issue on our end, but customers may experience some queueing until we catch up.

  • Investigating
    UTC
    Investigating

    We are currently investigating an issue where customers Github Actions are not being picked up by Runners across our regions.

EU Central Storage Cluster
  • Resolved
    UTC
    Resolved
    This incident has been resolved.
  • Monitoring
    UTC
    Monitoring
    We implemented a fix and are currently monitoring the result.
  • Investigating
    UTC
    Investigating

    We are currently investigating some service degradation with the eu-central storage cluster. You may see cache misses and slower disk speeds.

Delays in MacOS job adoption
  • Resolved
    UTC
    Resolved
    This incident has been resolved.
  • Monitoring
    UTC
    Monitoring

    Our team have mitigated this and are now seeing job pick up times return to normal as we monitor

  • Investigating
    UTC
    Investigating

    We're seeing a spike of MacOS jobs leading to a temporary capacity shortage. We're looking into re-balancing to reduce the delays

Github are reporting degraded performance for Actions and Webhooks
Job adoption delays
  • Update
    UTC
    Update
    This incident has been resolved.
  • Resolved
    UTC
    Resolved

    We implemented a fix and jobs are being picked up as normal. This incident is now closed.

  • Monitoring
    UTC
    Monitoring

    Our engineers are currently implementing a mitigation. Thank you for your patience.

  • Identified
    UTC
    Identified

    We have identified the issue as GitHub sending us a high cardinality of malformed webhooks missing critical pieces of information in the payloads. We are working on a patch to work around this as we wait for GitHub to fix the upstream issue.

  • Update
    UTC
    Update

    We believe this is related to an issue with Github Webhooks and are still investigating.

  • Investigating
    UTC
    Investigating

    We are receiving reports of job adoption delays, we are currently investigating.

May 2026

Delay in picking up jobs
  • Resolved
    UTC
    Resolved
    This incident has been resolved.
  • Update
    UTC
    Update

    We're continuing to monitor the situation. There's still a slight delay in jobs, but cancelling and rerunning the jobs should resolve the issue.

  • Monitoring
    UTC
    Monitoring

    Continuing to monitor improvements from deployment.

  • Identified
    UTC
    Identified

    We've deployed mitigations and are observing improvements.

  • Investigating
    UTC
    Investigating
    We are currently investigating this incident.
Github reporting degraded performance for API Requests, Git Operations and other services
  • Resolved
    UTC
    Resolved

    Github have resolved this incident.

  • Monitoring
    UTC
    Monitoring

    We will continue to monitor - https://www.githubstatus.com/incidents/xy1tt3hs572m

GitHub reporting degraded performance for Actions
  • Resolved
    UTC
    Resolved

    Github have resolved this incident.

  • Monitoring
    UTC
    Monitoring

    GitHub is reporting degraded performance with Actions as well as other services. This may have an impact on job adoption.
    We are monitoring this incident - https://www.githubstatus.com/incidents/gnftqj9htp0g

Delays in job adoption in eu-west
  • Resolved
    UTC
    Resolved
    This incident has been resolved.
  • Identified
    UTC
    Identified

    We are continuing to monitor this, as well as track an incident reported by with GitHub Actions which can be found here:

    https://www.githubstatus.com/incidents/g6ffrm0rfvz9

    We are continuing to re-balance in order to reduce delays.

  • Investigating
    UTC
    Investigating

    We're seeing a spike of jobs in the eu-west region leading to a temporary capacity shortage. We're looking into re-balancing to reduce the delays

Transparent caching degradaded in eu-west
  • Resolved
    UTC
    Resolved
    This incident has been resolved.
  • Monitoring
    UTC
    Monitoring

    The cluster expansion is complete. Our testing shows that cache interactions are no longer degraded. We're monitoring the cluster to confirm complete recovery.

  • Update
    UTC
    Update

    We're currently adding more capacity to the cluster. There will be a few minutes of slow cache interactions while the operation completes.

  • Identified
    UTC
    Identified

    We have identified a bad host in the fleet and are working on replacing it. Client cache interactions are recovering as we rollout this fix.

  • Monitoring
    UTC
    Monitoring

    We noticed a small uptick in transparent caching failures in our eu-west region, and have applied a potential fix.

Apr 2026

Degraded Actions performance in us-west
  • Resolved
    UTC
    Resolved
    This incident has been resolved.
  • Update
    UTC
    Update

    We are continuing to see recovery in us-west

  • Monitoring
    UTC
    Monitoring

    We are aware of some reports of slow Actions performance in our us-west region. A mitigation has been applied and we're now seeing signs of recovery

Blacksmith control plane DB maintenance
  • Resolved
    UTC
    Resolved

    This incident has been resolved.

  • Update
    UTC
    Update

    The re-queuing is still in progress however new jobs are being picked up without delay.

  • Monitoring
    UTC
    Monitoring

    We have completed the DB operation and currently re-queuing jobs. This may take ~15 minutes

  • Investigating
    UTC
    Investigating

    We are performing brief maintenance on our database. Customers may experience some delays with job adoption

Github Actions Degraded
  • Resolved
    UTC
    Resolved

    GitHub resolved their upstream outage and we are not seeing any customer impact.

  • Investigating
    UTC
    Investigating

    GitHub is reporting a GitHub Actions degradation, and we're monitoring the impact https://www.githubstatus.com/incidents/6bkk045k741z

Github Actions and Packages degraded
  • Resolved
    UTC
    Resolved

    This incident has been resolved by Github.

  • Monitoring
    UTC
    Monitoring

    It seems that there's still trailing effects from the incident and we're seeing queue times due to slow webhook delivery.

  • Resolved
    UTC
    Resolved

    This incident has been resolved by Github.

  • Investigating
    UTC
    Investigating

    We are currently monitoring the effects of a Github outage: https://www.githubstatus.com/incidents/3f30ycplyr7l

Sticky Disk degradation in the US-WEST cluster
  • Resolved
    UTC
    Resolved
    This incident has been resolved.
  • Monitoring
    UTC
    Monitoring

    We needed to perform an emergency maintenance operation in our storage cluster. We are observing the recovery and seeing docker build speeds improve.

  • Investigating
    UTC
    Investigating
    We are currently investigating this incident.

Apr 2026 to Jun 2026

Next