blacksmith - Notice history

All systems operational

99.94% uptime

eu-central Storage Cluster - Operational

99.92% uptime
May 2026 · 100.0%Jun · 100.0%Jul · 99.78%

Previous page

Next page

May 2026
Jun 2026
Jul 2026

us-west Storage Cluster - Operational

99.92% uptime
May 2026 · 100.0%Jun · 100.0%Jul · 99.78%

Previous page

Next page

May 2026
Jun 2026
Jul 2026

eu-west Storage Cluster - Operational

99.92% uptime
May 2026 · 100.0%Jun · 100.0%Jul · 99.78%

Previous page

Next page

May 2026
Jun 2026
Jul 2026

us-east Storage Cluster - Operational

100.0% uptime
May 2026 · 100.0%Jun · 100.0%Jul · 100.0%

Previous page

Next page

May 2026
Jun 2026
Jul 2026

API - Operational

100.0% uptime
May 2026 · 100.0%Jun · 100.0%Jul · 100.0%

Previous page

Next page

May 2026
Jun 2026
Jul 2026
100.0% uptime

https://blacksmith.sh - Operational

100.0% uptime
May 2026 · 100.0%Jun · 100.0%Jul · 100.0%

Previous page

Next page

May 2026
Jun 2026
Jul 2026

Github → Actions - Operational

Github → API Requests - Operational

Github → Webhooks - Operational

Codesmith - Operational

100.0% uptime
May 2026 · 100.0%Jun · 100.0%Jul · 99.99%

Previous page

Next page

May 2026
Jun 2026
Jul 2026

Dashboard - Operational

100.0% uptime
May 2026 · 100.0%Jun · 100.0%Jul · 100.0%

Previous page

Next page

May 2026
Jun 2026
Jul 2026

Notice history

View current status

Jul 2026

JUL30
Runtime Build Caching
CompletedMaintenance1 hour 43 minutes
  • Completed
    July 30, 2026 at 3:13 AMUTC
    Completed
    July 30, 2026 at 3:13 AMUTC

    Maintenance has completed successfully. Runtime Build Caching has been fully restored and is operating normally.

  • In progress
    July 30, 2026 at 1:30 AMUTC
    In progress
    July 30, 2026 at 1:30 AMUTC
    Maintenance is now in progress
  • Planned
    July 30, 2026 at 1:15 AMUTC
    Planned
    July 30, 2026 at 1:15 AMUTC

    We will be performing maintenance on our Runtime Build Caching infrastructure. Jobs will continue to run, but some may run without remote caching and take longer than usual during this window. No customer action is required. We expect maintenance to last approximately two hours.

Increased Error Rate on Codesmith Sandbox Startup
ResolvedMajor outage26 minutes
  • Resolved
    UTC
    Resolved
    This incident has been resolved.
  • Monitoring
    UTC
    Monitoring

    The upstream issue is resolved, we are monitoring the sandbox creation rate recovering.

  • Identified
    UTC
    Identified

    We're seeing increased error rates from an upstream provider in spinning up sandbox environments in Codesmith. We are looking into remediation options.

Jobs not being adopted in all regions
ResolvedPartial outage33 minutes
  • Resolved
    UTC
    Resolved

    This incident has been resolved, and job pickup has returned to normal.

  • Update
    UTC
    Update

    We have requeued the webhooks that were not processed during the incident, and any affected jobs should start shortly.

  • Monitoring
    UTC
    Monitoring

    We have deployed a fix and are monitoring recovery, and will provide another update within the next 30 minutes.

  • Investigating
    UTC
    Investigating

    We are seeing webhooks not being accepted by our control plane resulting in jobs not being adopted. We are currently investigating.

Sticky disk degradation in EU-West
ResolvedDegraded performance43 minutes
  • Resolved
    UTC
    Resolved
    This incident has been resolved.
  • Monitoring
    UTC
    Monitoring
    We implemented a fix and are currently monitoring the result.
  • Investigating
    UTC
    Investigating

    We're seeing large volumes of traffic in our storage cluster in EU-West and this is causing a number of sticky disk related requests to time out and are actively investigating.

Extended Queueing in Action Runners
ResolvedDegraded performance20 minutes
  • Resolved
    UTC
    Resolved

    We have implemented some mitigations and queue times are back to normal. The incident has been resolved.

  • Investigating
    UTC
    Investigating

    We are currently experiencing elevated tail latencies for certain customers due to job prioritization. We are noticing jobs are taking upwards of 10m to adopt in a few cases. We are looking at options to alleviate queueing.

Jun 2026

Runners hitting Github rate limiting
ResolvedMajor outage1 hour 11 minutes
  • Resolved
    UTC
    Resolved
    This incident has been resolved.
  • Monitoring
    UTC
    Monitoring

    Queueing improving and currently monitoring.

  • Identified
    UTC
    Identified

    We are have mitigated the issue on our end, but customers may experience some queueing until we catch up.

  • Investigating
    UTC
    Investigating

    We are currently investigating an issue where customers Github Actions are not being picked up by Runners across our regions.

EU Central Storage Cluster
ResolvedDegraded performance54 minutes
  • Resolved
    UTC
    Resolved
    This incident has been resolved.
  • Monitoring
    UTC
    Monitoring
    We implemented a fix and are currently monitoring the result.
  • Investigating
    UTC
    Investigating

    We are currently investigating some service degradation with the eu-central storage cluster. You may see cache misses and slower disk speeds.

Delays in MacOS job adoption
ResolvedDegraded performance53 minutes
  • Resolved
    UTC
    Resolved
    This incident has been resolved.
  • Monitoring
    UTC
    Monitoring

    Our team have mitigated this and are now seeing job pick up times return to normal as we monitor

  • Investigating
    UTC
    Investigating

    We're seeing a spike of MacOS jobs leading to a temporary capacity shortage. We're looking into re-balancing to reduce the delays

Github are reporting degraded performance for Actions and Webhooks
ResolvedDegraded performance7 minutes
Job adoption delays
ResolvedMajor outage1 hour 3 minutes
  • Update
    UTC
    Update
    This incident has been resolved.
  • Resolved
    UTC
    Resolved

    We implemented a fix and jobs are being picked up as normal. This incident is now closed.

  • Monitoring
    UTC
    Monitoring

    Our engineers are currently implementing a mitigation. Thank you for your patience.

  • Identified
    UTC
    Identified

    We have identified the issue as GitHub sending us a high cardinality of malformed webhooks missing critical pieces of information in the payloads. We are working on a patch to work around this as we wait for GitHub to fix the upstream issue.

  • Update
    UTC
    Update

    We believe this is related to an issue with Github Webhooks and are still investigating.

  • Investigating
    UTC
    Investigating

    We are receiving reports of job adoption delays, we are currently investigating.

May 2026

Delay in picking up jobs
ResolvedMajor outage5 hours 16 minutes
  • Resolved
    UTC
    Resolved
    This incident has been resolved.
  • Update
    UTC
    Update

    We're continuing to monitor the situation. There's still a slight delay in jobs, but cancelling and rerunning the jobs should resolve the issue.

  • Monitoring
    UTC
    Monitoring

    Continuing to monitor improvements from deployment.

  • Identified
    UTC
    Identified

    We've deployed mitigations and are observing improvements.

  • Investigating
    UTC
    Investigating
    We are currently investigating this incident.
Github reporting degraded performance for API Requests, Git Operations and other services
ResolvedMajor outage32 minutes
  • Resolved
    UTC
    Resolved

    Github have resolved this incident.

  • Monitoring
    UTC
    Monitoring

    We will continue to monitor - https://www.githubstatus.com/incidents/xy1tt3hs572m

GitHub reporting degraded performance for Actions
ResolvedDegraded performance2 hours 1 minute
  • Resolved
    UTC
    Resolved

    Github have resolved this incident.

  • Monitoring
    UTC
    Monitoring

    GitHub is reporting degraded performance with Actions as well as other services. This may have an impact on job adoption.
    We are monitoring this incident - https://www.githubstatus.com/incidents/gnftqj9htp0g

Delays in job adoption in eu-west
ResolvedDegraded performance2 hours 39 minutes
  • Resolved
    UTC
    Resolved
    This incident has been resolved.
  • Identified
    UTC
    Identified

    We are continuing to monitor this, as well as track an incident reported by with GitHub Actions which can be found here:

    https://www.githubstatus.com/incidents/g6ffrm0rfvz9

    We are continuing to re-balance in order to reduce delays.

  • Investigating
    UTC
    Investigating

    We're seeing a spike of jobs in the eu-west region leading to a temporary capacity shortage. We're looking into re-balancing to reduce the delays

Transparent caching degradaded in eu-west
ResolvedDegraded performance12 hours 58 minutes
  • Resolved
    UTC
    Resolved
    This incident has been resolved.
  • Monitoring
    UTC
    Monitoring

    The cluster expansion is complete. Our testing shows that cache interactions are no longer degraded. We're monitoring the cluster to confirm complete recovery.

  • Update
    UTC
    Update

    We're currently adding more capacity to the cluster. There will be a few minutes of slow cache interactions while the operation completes.

  • Identified
    UTC
    Identified

    We have identified a bad host in the fleet and are working on replacing it. Client cache interactions are recovering as we rollout this fix.

  • Monitoring
    UTC
    Monitoring

    We noticed a small uptick in transparent caching failures in our eu-west region, and have applied a potential fix.

Previous

May 2026 to Jul 2026

Next