blacksmith - Notice history

All systems operational

100% - uptime

eu-central Storage Cluster - Operational

100% - uptime
Jun 2026 · 100.0%Jul · 100.0%Aug · 99.98%

Previous page

Next page

Jun 2026
Jul 2026
Aug 2026

us-west Storage Cluster - Operational

100% - uptime
Jun 2026 · 100.0%Jul · 100.0%Aug · 99.97%

Previous page

Next page

Jun 2026
Jul 2026
Aug 2026

eu-west Storage Cluster - Operational

100% - uptime
Jun 2026 · 100.0%Jul · 100.0%Aug · 99.98%

Previous page

Next page

Jun 2026
Jul 2026
Aug 2026

Actions Cache - Operational

100% - uptime
Jun 2026 · 100.0%Jul · 99.84%Aug · 99.98%

Previous page

Next page

Jun 2026
Jul 2026
Aug 2026

Runtime Build Caching - Operational

API - Operational

100% - uptime
Jun 2026 · 100.0%Jul · 100.0%Aug · 100.0%

Previous page

Next page

Jun 2026
Jul 2026
Aug 2026

Website - Operational

100% - uptime
Jun 2026 · 100.0%Jul · 99.94%Aug · 99.98%

Previous page

Next page

Jun 2026
Jul 2026
Aug 2026

Github → Actions - Operational

Github → API Requests - Operational

Github → Webhooks - Operational

https://blacksmith.sh - Operational

100% - uptime
Jun 2026 · 100.0%Jul · 100.0%Aug · 99.98%

Previous page

Next page

Jun 2026
Jul 2026
Aug 2026

Codesmith - Operational

100% - uptime
Jun 2026 · 100.0%Jul · 99.99%Aug · 99.98%

Previous page

Next page

Jun 2026
Jul 2026
Aug 2026

Website - Operational

100% - uptime
Jun 2026 · 100.0%Jul · 100.0%Aug · 100.0%

Previous page

Next page

Jun 2026
Jul 2026
Aug 2026

Notice history

Aug 2026

Github → Actions experiencing degraded performance
  • Resolved
    UTC
    Resolved

    GitHub job success and adoption rates are now back at normal levels. Jobs that were not adopted during the incident are being requeued by our team and should be picked up shortly.

  • Monitoring
    UTC
    Monitoring

    GitHub has reported degraded performance for Actions. Jobs may be delayed to start, or fail during the set up step. We are monitoring this incident. https://www.githubstatus.com/incidents/qcvjkzcs7j74

Jobs not getting picked up
  • Resolved
    UTC
    Resolved
    This incident has been resolved.
  • Update
    UTC
    Update

    Job adoption is now fully operational.

  • Update
    UTC
    Update

    There is still a delay in job adoption times as we continue to recover.

  • Monitoring
    UTC
    Monitoring

    Jobs adoption has recovered and are operational.

  • Identified
    UTC
    Identified

    We've mitigated the root cause and jobs are resuming to run. Some jobs may still be delayed to start as we catch up with the job backlog.

  • Investigating
    UTC
    Investigating

    We're currently experiencing an outage of our control plane. GitHub job adoption and execution are affected, as well as dashboard access.

Elevated error rates in Sticky Disk and Actions Cache requests
  • Resolved
    UTC
    Resolved
    This incident has been resolved.
  • Update
    UTC
    Update

    We are still monitoring recovery.

  • Monitoring
    UTC
    Monitoring
    We implemented a fix and are currently monitoring the result.
  • Investigating
    UTC
    Investigating

    We are seeing elevated error rates for Sticky Disk and Actions Cache requests. We are investigating.

Delay in Job Adoption
  • Resolved
    UTC
    Resolved
    This incident has been resolved.
  • Investigating
    UTC
    Investigating

    We're seeing a number of cases where jobs may not be adopted promptly. We're currently investigating.

US-West storage cluster degraded
  • Resolved
    UTC
    Resolved
    This incident has been resolved.
  • Monitoring
    UTC
    Monitoring

    We've applied a fix and are seeing failure rates and latency starting to come down.

  • Investigating
    UTC
    Investigating

    We are seeing some failures with sticky disk availability and increased latency with the US-West storage cluster.

Jul 2026

Runtime Build Caching
  • Completed
    July 30, 2026 at 3:13 AMUTC
    Completed
    July 30, 2026 at 3:13 AMUTC

    Maintenance has completed successfully. Runtime Build Caching has been fully restored and is operating normally.

  • In progress
    July 30, 2026 at 1:30 AMUTC
    In progress
    July 30, 2026 at 1:30 AMUTC
    Maintenance is now in progress
  • Planned
    July 30, 2026 at 1:15 AMUTC
    Planned
    July 30, 2026 at 1:15 AMUTC

    We will be performing maintenance on our Runtime Build Caching infrastructure. Jobs will continue to run, but some may run without remote caching and take longer than usual during this window. No customer action is required. We expect maintenance to last approximately two hours.

Increased Error Rate on Codesmith Sandbox Startup
  • Resolved
    UTC
    Resolved
    This incident has been resolved.
  • Monitoring
    UTC
    Monitoring

    The upstream issue is resolved, we are monitoring the sandbox creation rate recovering.

  • Identified
    UTC
    Identified

    We're seeing increased error rates from an upstream provider in spinning up sandbox environments in Codesmith. We are looking into remediation options.

Jobs not being adopted in all regions
  • Resolved
    UTC
    Resolved

    This incident has been resolved, and job pickup has returned to normal.

  • Update
    UTC
    Update

    We have requeued the webhooks that were not processed during the incident, and any affected jobs should start shortly.

  • Monitoring
    UTC
    Monitoring

    We have deployed a fix and are monitoring recovery, and will provide another update within the next 30 minutes.

  • Investigating
    UTC
    Investigating

    We are seeing webhooks not being accepted by our control plane resulting in jobs not being adopted. We are currently investigating.

Sticky disk degradation in EU-West
  • Resolved
    UTC
    Resolved
    This incident has been resolved.
  • Monitoring
    UTC
    Monitoring
    We implemented a fix and are currently monitoring the result.
  • Investigating
    UTC
    Investigating

    We're seeing large volumes of traffic in our storage cluster in EU-West and this is causing a number of sticky disk related requests to time out and are actively investigating.

Extended Queueing in Action Runners
  • Resolved
    UTC
    Resolved

    We have implemented some mitigations and queue times are back to normal. The incident has been resolved.

  • Investigating
    UTC
    Investigating

    We are currently experiencing elevated tail latencies for certain customers due to job prioritization. We are noticing jobs are taking upwards of 10m to adopt in a few cases. We are looking at options to alleviate queueing.

Jun 2026

Runners hitting Github rate limiting
  • Resolved
    UTC
    Resolved
    This incident has been resolved.
  • Monitoring
    UTC
    Monitoring

    Queueing improving and currently monitoring.

  • Identified
    UTC
    Identified

    We are have mitigated the issue on our end, but customers may experience some queueing until we catch up.

  • Investigating
    UTC
    Investigating

    We are currently investigating an issue where customers Github Actions are not being picked up by Runners across our regions.

EU Central Storage Cluster
  • Resolved
    UTC
    Resolved
    This incident has been resolved.
  • Monitoring
    UTC
    Monitoring
    We implemented a fix and are currently monitoring the result.
  • Investigating
    UTC
    Investigating

    We are currently investigating some service degradation with the eu-central storage cluster. You may see cache misses and slower disk speeds.

Delays in MacOS job adoption
  • Resolved
    UTC
    Resolved
    This incident has been resolved.
  • Monitoring
    UTC
    Monitoring

    Our team have mitigated this and are now seeing job pick up times return to normal as we monitor

  • Investigating
    UTC
    Investigating

    We're seeing a spike of MacOS jobs leading to a temporary capacity shortage. We're looking into re-balancing to reduce the delays

Github are reporting degraded performance for Actions and Webhooks
Job adoption delays
  • Update
    UTC
    Update
    This incident has been resolved.
  • Resolved
    UTC
    Resolved

    We implemented a fix and jobs are being picked up as normal. This incident is now closed.

  • Monitoring
    UTC
    Monitoring

    Our engineers are currently implementing a mitigation. Thank you for your patience.

  • Identified
    UTC
    Identified

    We have identified the issue as GitHub sending us a high cardinality of malformed webhooks missing critical pieces of information in the payloads. We are working on a patch to work around this as we wait for GitHub to fix the upstream issue.

  • Update
    UTC
    Update

    We believe this is related to an issue with Github Webhooks and are still investigating.

  • Investigating
    UTC
    Investigating

    We are receiving reports of job adoption delays, we are currently investigating.

Jun 2026 to Aug 2026

Next