Status | Blacksmith - Notice history

All systems operational

100% - uptime

eu-central Storage Cluster - Operational

100% - uptime
May 2026 · 100.0%Jun · 100.0%Jul · 100.0%

Previous page

Next page

May 2026
Jun 2026
Jul 2026

us-west Storage Cluster - Operational

100% - uptime
May 2026 · 100.0%Jun · 100.0%Jul · 100.0%

Previous page

Next page

May 2026
Jun 2026
Jul 2026

eu-west Storage Cluster - Operational

100% - uptime
May 2026 · 100.0%Jun · 100.0%Jul · 100.0%

Previous page

Next page

May 2026
Jun 2026
Jul 2026

Actions Cache - Operational

100% - uptime
May 2026 · 100.0%Jun · 100.0%Jul · 99.84%

Previous page

Next page

May 2026
Jun 2026
Jul 2026

API - Operational

100% - uptime
May 2026 · 100.0%Jun · 100.0%Jul · 100.0%

Previous page

Next page

May 2026
Jun 2026
Jul 2026

Website - Operational

100% - uptime
May 2026 · 100.0%Jun · 100.0%Jul · 99.94%

Previous page

Next page

May 2026
Jun 2026
Jul 2026

Github → Actions - Operational

Github → API Requests - Operational

Github → Webhooks - Operational

Website - Operational

100% - uptime
May 2026 · 100.0%Jun · 100.0%Jul · 100.0%

Previous page

Next page

May 2026
Jun 2026
Jul 2026

https://blacksmith.sh - Operational

100% - uptime
May 2026 · 100.0%Jun · 100.0%Jul · 100.0%

Previous page

Next page

May 2026
Jun 2026
Jul 2026

Notice history

Jul 2026

GitHub Outage Affecting Jobs
  • Resolved
    UTC
    Resolved
    This incident has been resolved.
  • Monitoring
    UTC
    Monitoring

    Job executions have returned to normal. We are monitoring the recovery.

  • Update
    UTC
    Update

    Requests to GitHub API are now succeeding. There maybe some some delays in job adoption as we work through the large backlog of jobs that did not run during the incident.

  • Update
    UTC
    Update

    We are starting to see signs of recovery of upstream API errors and jobs have resumed running. We are continuing to monitor.

  • Update
    UTC
    Update

    We are continuing to see errors from GitHub when trying to provision CI jobs. They are continuing to investigate their incident.

    https://www.githubstatus.com/incidents/8vfyvq16hzh9

  • Identified
    UTC
    Identified

    GitHub has opened an incident which we are seeing is affecting jobs getting picked up.

    https://www.githubstatus.com/incidents/8vfyvq16hzh9

Intermittent dashboard loading and CI job errors due to GitHub outage
  • Resolved
    UTC
    Resolved

    The upstream GitHub incident has been resolved: https://www.githubstatus.com/

    We will continue to monitor GitHub API Stability.

  • Monitoring
    UTC
    Monitoring

    We are seeing errors affecting the availability to log into the Blacksmith dashboard due to upstream calls to GitHub. There are widespread reports of GitHub API failures: https://www.githubstatus.com/ | https://updog.ai/status/github

    We are seeing this impact CI jobs that interact with GitHub APIs such as downloading actions, uploading artifacts, etc.

  • Identified
    UTC
    Identified

    We are seeing errors affecting the availability to log into the Blacksmith dashboard due to upstream calls to GitHub. We are not currently seeing any impact to the execution of CI jobs.

  • Investigating
    UTC
    Investigating
    We are currently investigating this incident.
Stickydisk Storage cluster maintenance
  • Completed
    July 13, 2026 at 12:50 AMUTC
    Completed
    July 13, 2026 at 12:50 AMUTC
    Maintenance has completed successfully.
  • In progress
    July 12, 2026 at 8:00 PMUTC
    In progress
    July 12, 2026 at 8:00 PMUTC
    Maintenance is now in progress
  • Planned
    July 12, 2026 at 3:05 PMUTC
    Planned
    July 12, 2026 at 3:05 PMUTC

    We will be performing Ceph major version upgrade. No downtime expected.

Stickydisk Storage clusters maintenance
  • Completed
    July 12, 2026 at 3:03 PMUTC
    Completed
    July 12, 2026 at 3:03 PMUTC
    Maintenance has completed successfully.
  • In progress
    July 12, 2026 at 12:30 PMUTC
    In progress
    July 12, 2026 at 12:30 PMUTC
    Maintenance is now in progress
  • Planned
    July 12, 2026 at 12:25 PMUTC
    Planned
    July 12, 2026 at 12:25 PMUTC

    We will be performing Ceph major version upgrade. No downtime expected.

Sticky disk attach failures in us-west
  • Resolved
    UTC
    Resolved

    This incident is resolved. The issue was identified and fixed. Sticky disk attachments in us-west were affected between 20:40 and 21:15 UTC.

  • Investigating
    UTC
    Investigating

    We are currently experiencing an issue in our us-west region where sticky disks are failing to attach. Sticky disks back several of our caching features, so customers running jobs in us-west may be affected across Docker container caching, Docker build caching, and Git repository caching. Affected jobs will still run, but will fall back to uncached behavior and may take longer than usual. Runners in other regions are unaffected. We are actively investigating and will post updates as we make progress.

Jun 2026

Runners hitting Github rate limiting
  • Resolved
    UTC
    Resolved
    This incident has been resolved.
  • Monitoring
    UTC
    Monitoring

    Queueing improving and currently monitoring.

  • Identified
    UTC
    Identified

    We are have mitigated the issue on our end, but customers may experience some queueing until we catch up.

  • Investigating
    UTC
    Investigating

    We are currently investigating an issue where customers Github Actions are not being picked up by Runners across our regions.

EU Central Storage Cluster
  • Resolved
    UTC
    Resolved
    This incident has been resolved.
  • Monitoring
    UTC
    Monitoring
    We implemented a fix and are currently monitoring the result.
  • Investigating
    UTC
    Investigating

    We are currently investigating some service degradation with the eu-central storage cluster. You may see cache misses and slower disk speeds.

Delays in MacOS job adoption
  • Resolved
    UTC
    Resolved
    This incident has been resolved.
  • Monitoring
    UTC
    Monitoring

    Our team have mitigated this and are now seeing job pick up times return to normal as we monitor

  • Investigating
    UTC
    Investigating

    We're seeing a spike of MacOS jobs leading to a temporary capacity shortage. We're looking into re-balancing to reduce the delays

Github are reporting degraded performance for Actions and Webhooks
Job adoption delays
  • Update
    UTC
    Update
    This incident has been resolved.
  • Resolved
    UTC
    Resolved

    We implemented a fix and jobs are being picked up as normal. This incident is now closed.

  • Monitoring
    UTC
    Monitoring

    Our engineers are currently implementing a mitigation. Thank you for your patience.

  • Identified
    UTC
    Identified

    We have identified the issue as GitHub sending us a high cardinality of malformed webhooks missing critical pieces of information in the payloads. We are working on a patch to work around this as we wait for GitHub to fix the upstream issue.

  • Update
    UTC
    Update

    We believe this is related to an issue with Github Webhooks and are still investigating.

  • Investigating
    UTC
    Investigating

    We are receiving reports of job adoption delays, we are currently investigating.

May 2026

Delay in picking up jobs
  • Resolved
    UTC
    Resolved
    This incident has been resolved.
  • Update
    UTC
    Update

    We're continuing to monitor the situation. There's still a slight delay in jobs, but cancelling and rerunning the jobs should resolve the issue.

  • Monitoring
    UTC
    Monitoring

    Continuing to monitor improvements from deployment.

  • Identified
    UTC
    Identified

    We've deployed mitigations and are observing improvements.

  • Investigating
    UTC
    Investigating
    We are currently investigating this incident.
Github reporting degraded performance for API Requests, Git Operations and other services
  • Resolved
    UTC
    Resolved

    Github have resolved this incident.

  • Monitoring
    UTC
    Monitoring

    We will continue to monitor - https://www.githubstatus.com/incidents/xy1tt3hs572m

GitHub reporting degraded performance for Actions
  • Resolved
    UTC
    Resolved

    Github have resolved this incident.

  • Monitoring
    UTC
    Monitoring

    GitHub is reporting degraded performance with Actions as well as other services. This may have an impact on job adoption.
    We are monitoring this incident - https://www.githubstatus.com/incidents/gnftqj9htp0g

Delays in job adoption in eu-west
  • Resolved
    UTC
    Resolved
    This incident has been resolved.
  • Identified
    UTC
    Identified

    We are continuing to monitor this, as well as track an incident reported by with GitHub Actions which can be found here:

    https://www.githubstatus.com/incidents/g6ffrm0rfvz9

    We are continuing to re-balance in order to reduce delays.

  • Investigating
    UTC
    Investigating

    We're seeing a spike of jobs in the eu-west region leading to a temporary capacity shortage. We're looking into re-balancing to reduce the delays

Transparent caching degradaded in eu-west
  • Resolved
    UTC
    Resolved
    This incident has been resolved.
  • Monitoring
    UTC
    Monitoring

    The cluster expansion is complete. Our testing shows that cache interactions are no longer degraded. We're monitoring the cluster to confirm complete recovery.

  • Update
    UTC
    Update

    We're currently adding more capacity to the cluster. There will be a few minutes of slow cache interactions while the operation completes.

  • Identified
    UTC
    Identified

    We have identified a bad host in the fleet and are working on replacing it. Client cache interactions are recovering as we rollout this fix.

  • Monitoring
    UTC
    Monitoring

    We noticed a small uptick in transparent caching failures in our eu-west region, and have applied a potential fix.

May 2026 to Jul 2026

Next