blacksmith - Notice history

All systems operational

99.94% uptime

eu-central Storage Cluster - Operational

99.92% uptime
May 2026 · 100.0%Jun · 100.0%Jul · 99.78%

Previous page

Next page

May 2026
Jun 2026
Jul 2026

us-west Storage Cluster - Operational

99.92% uptime
May 2026 · 100.0%Jun · 100.0%Jul · 99.78%

Previous page

Next page

May 2026
Jun 2026
Jul 2026

eu-west Storage Cluster - Operational

99.92% uptime
May 2026 · 100.0%Jun · 100.0%Jul · 99.78%

Previous page

Next page

May 2026
Jun 2026
Jul 2026

us-east Storage Cluster - Operational

100.0% uptime
May 2026 · 100.0%Jun · 100.0%Jul · 100.0%

Previous page

Next page

May 2026
Jun 2026
Jul 2026

API - Operational

100.0% uptime
May 2026 · 100.0%Jun · 100.0%Jul · 100.0%

Previous page

Next page

May 2026
Jun 2026
Jul 2026
100.0% uptime

https://blacksmith.sh - Operational

100.0% uptime
May 2026 · 100.0%Jun · 100.0%Jul · 100.0%

Previous page

Next page

May 2026
Jun 2026
Jul 2026

Github → Actions - Operational

Github → API Requests - Operational

Github → Webhooks - Operational

Codesmith - Operational

100.0% uptime
May 2026 · 100.0%Jun · 100.0%Jul · 99.99%

Previous page

Next page

May 2026
Jun 2026
Jul 2026

Dashboard - Operational

100.0% uptime
May 2026 · 100.0%Jun · 100.0%Jul · 100.0%

Previous page

Next page

May 2026
Jun 2026
Jul 2026

Notice history

View current status

Jul 2026

JUL30
Runtime Build Caching
CompletedMaintenance1 hour 43 minutes
  • CompletedJuly 30, 2026 at 3:13 AMUTC

    Maintenance has completed successfully. Runtime Build Caching has been fully restored and is operating normally.

  • In progressJuly 30, 2026 at 1:30 AMUTC
    Maintenance is now in progress
  • PlannedJuly 30, 2026 at 1:15 AMUTC

    We will be performing maintenance on our Runtime Build Caching infrastructure. Jobs will continue to run, but some may run without remote caching and take longer than usual during this window. No customer action is required. We expect maintenance to last approximately two hours.

Increased Error Rate on Codesmith Sandbox Startup
ResolvedMajor outage26 minutes
  • ResolvedUTC
    This incident has been resolved.
  • MonitoringUTC

    The upstream issue is resolved, we are monitoring the sandbox creation rate recovering.

  • IdentifiedUTC

    We're seeing increased error rates from an upstream provider in spinning up sandbox environments in Codesmith. We are looking into remediation options.

Jobs not being adopted in all regions
ResolvedPartial outage33 minutes
  • ResolvedUTC

    This incident has been resolved, and job pickup has returned to normal.

  • UpdateUTC

    We have requeued the webhooks that were not processed during the incident, and any affected jobs should start shortly.

  • MonitoringUTC

    We have deployed a fix and are monitoring recovery, and will provide another update within the next 30 minutes.

  • InvestigatingUTC

    We are seeing webhooks not being accepted by our control plane resulting in jobs not being adopted. We are currently investigating.

Sticky disk degradation in EU-West
ResolvedDegraded performance43 minutes
  • ResolvedUTC
    This incident has been resolved.
  • MonitoringUTC
    We implemented a fix and are currently monitoring the result.
  • InvestigatingUTC

    We're seeing large volumes of traffic in our storage cluster in EU-West and this is causing a number of sticky disk related requests to time out and are actively investigating.

Extended Queueing in Action Runners
ResolvedDegraded performance20 minutes
  • ResolvedUTC

    We have implemented some mitigations and queue times are back to normal. The incident has been resolved.

  • InvestigatingUTC

    We are currently experiencing elevated tail latencies for certain customers due to job prioritization. We are noticing jobs are taking upwards of 10m to adopt in a few cases. We are looking at options to alleviate queueing.

Jun 2026

Runners hitting Github rate limiting
ResolvedMajor outage1 hour 11 minutes
  • ResolvedUTC
    This incident has been resolved.
  • MonitoringUTC

    Queueing improving and currently monitoring.

  • IdentifiedUTC

    We are have mitigated the issue on our end, but customers may experience some queueing until we catch up.

  • InvestigatingUTC

    We are currently investigating an issue where customers Github Actions are not being picked up by Runners across our regions.

EU Central Storage Cluster
ResolvedDegraded performance54 minutes
  • ResolvedUTC
    This incident has been resolved.
  • MonitoringUTC
    We implemented a fix and are currently monitoring the result.
  • InvestigatingUTC

    We are currently investigating some service degradation with the eu-central storage cluster. You may see cache misses and slower disk speeds.

Delays in MacOS job adoption
ResolvedDegraded performance53 minutes
  • ResolvedUTC
    This incident has been resolved.
  • MonitoringUTC

    Our team have mitigated this and are now seeing job pick up times return to normal as we monitor

  • InvestigatingUTC

    We're seeing a spike of MacOS jobs leading to a temporary capacity shortage. We're looking into re-balancing to reduce the delays

Github are reporting degraded performance for Actions and Webhooks
ResolvedDegraded performance7 minutes
Job adoption delays
ResolvedMajor outage1 hour 3 minutes
  • UpdateUTC
    This incident has been resolved.
  • ResolvedUTC

    We implemented a fix and jobs are being picked up as normal. This incident is now closed.

  • MonitoringUTC

    Our engineers are currently implementing a mitigation. Thank you for your patience.

  • IdentifiedUTC

    We have identified the issue as GitHub sending us a high cardinality of malformed webhooks missing critical pieces of information in the payloads. We are working on a patch to work around this as we wait for GitHub to fix the upstream issue.

  • UpdateUTC

    We believe this is related to an issue with Github Webhooks and are still investigating.

  • InvestigatingUTC

    We are receiving reports of job adoption delays, we are currently investigating.

May 2026

Delay in picking up jobs
ResolvedMajor outage5 hours 16 minutes
  • ResolvedUTC
    This incident has been resolved.
  • UpdateUTC

    We're continuing to monitor the situation. There's still a slight delay in jobs, but cancelling and rerunning the jobs should resolve the issue.

  • MonitoringUTC

    Continuing to monitor improvements from deployment.

  • IdentifiedUTC

    We've deployed mitigations and are observing improvements.

  • InvestigatingUTC
    We are currently investigating this incident.
Github reporting degraded performance for API Requests, Git Operations and other services
ResolvedMajor outage32 minutes
GitHub reporting degraded performance for Actions
ResolvedDegraded performance2 hours 1 minute
  • ResolvedUTC

    Github have resolved this incident.

  • MonitoringUTC

    GitHub is reporting degraded performance with Actions as well as other services. This may have an impact on job adoption.
    We are monitoring this incident - https://www.githubstatus.com/incidents/gnftqj9htp0g

Delays in job adoption in eu-west
ResolvedDegraded performance2 hours 39 minutes
  • ResolvedUTC
    This incident has been resolved.
  • IdentifiedUTC

    We are continuing to monitor this, as well as track an incident reported by with GitHub Actions which can be found here:

    https://www.githubstatus.com/incidents/g6ffrm0rfvz9

    We are continuing to re-balance in order to reduce delays.

  • InvestigatingUTC

    We're seeing a spike of jobs in the eu-west region leading to a temporary capacity shortage. We're looking into re-balancing to reduce the delays

Transparent caching degradaded in eu-west
ResolvedDegraded performance12 hours 58 minutes
  • ResolvedUTC
    This incident has been resolved.
  • MonitoringUTC

    The cluster expansion is complete. Our testing shows that cache interactions are no longer degraded. We're monitoring the cluster to confirm complete recovery.

  • UpdateUTC

    We're currently adding more capacity to the cluster. There will be a few minutes of slow cache interactions while the operation completes.

  • IdentifiedUTC

    We have identified a bad host in the fleet and are working on replacing it. Client cache interactions are recovering as we rollout this fix.

  • MonitoringUTC

    We noticed a small uptick in transparent caching failures in our eu-west region, and have applied a potential fix.

Previous

May 2026 to Jul 2026

Next