Duration: 1 hour

Affected Components: us-west Storage Cluster

Oct 10, 00:40:00 GMT+0
Completed - Maintenance has completed successfully.

Oct 9, 23:40:01 GMT+0
Identified - Maintenance is now in progress.

Oct 9, 23:31:10 GMT+0
Identified - We are migrating stickydisk metadata to a new database backend. No downtime expected..

Duration: 1 hour

Affected Components: eu-west Storage Cluster

Oct 5, 23:15:00 GMT+0
Completed - Maintenance has completed successfully.

Oct 5, 22:15:01 GMT+0
Identified - Maintenance is now in progress.

Oct 5, 22:12:59 GMT+0
Identified - We are migrating stickydisk metadata to a new database backend. No downtime expected..

Duration: 3 hours and 37 minutes

Affected Components: eu-west ARM, eu-west x86

Oct 5, 03:37:25 GMT+0
Completed - Maintenance has completed successfully..

Oct 5, 00:00:01 GMT+0
Identified - Maintenance is now in progress.

Oct 1, 15:31:46 GMT+0
Identified - We are planning for a scheduled network maintenance during that time. You may see a slight increase in Github request latencies during this period if your jobs are running in the EU-West region..

Duration: 1 hour

Affected Components: us-east Storage Cluster

Oct 2, 21:30:00 GMT+0
Completed - Maintenance has completed successfully.

Oct 2, 20:30:01 GMT+0
Identified - Maintenance is now in progress.

Oct 2, 20:23:46 GMT+0
Identified - We are migrating stickydisk metadata to a new database backend. No downtime expected..

Duration: 1 hour

Affected Components: eu-central Storage Cluster

Sep 30, 22:00:00 GMT+0
Completed - Maintenance has completed successfully.

Sep 30, 21:00:01 GMT+0
Identified - Maintenance is now in progress.

Sep 30, 20:49:59 GMT+0
Identified - We are migrating stickydisk metadata to a new database backend. No downtime expected..

Duration: 1 hour and 54 minutes

Affected Components: us-east x86, US East Cache, eu-west Storage Cluster, eu-west ARM, eu-central x86, eu-central Storage Cluster, us-west Storage Cluster, EU Central Cache, us-west x86, eu-west x86, EU West Cache, eu-west Storage Cluster, eu-west Storage Cluster, eu-central Storage Cluster, eu-central Storage Cluster, US West Cache, us-central MacOS, eu-central ARM, us-west Storage Cluster, us-west Storage Cluster, us-west ARM, us-east Storage Cluster, us-east Storage Cluster, us-east Storage Cluster, US Central Cache

Sep 29, 23:04:31 GMT+0
Resolved - This incident has been resolved..

Sep 29, 22:42:09 GMT+0
Monitoring - This incident is resolved. Job start times and cache operations returned to normal in all regions, and we are continuing to monitor. Jobs that failed during the incident can be re-run..

Sep 29, 22:20:10 GMT+0
Monitoring - Caching has recovered, and job queue times across EU regions have recovered. Queue times for larger jobs (16 and 32 vcpu jobs) in US west and US east continue to remain elevated. We are continuing to monitor for any regressions..

Sep 29, 21:43:21 GMT+0
Monitoring - Our mitigation has taken effect and job start times and cache operations are improving. Customers may still see some delayed job starts and cache failures while recovery completes. We will provide an update within the next 30 minutes..

Sep 29, 21:28:22 GMT+0
Identified - We have applied a fix and are monitoring its effect. Customers may still see delayed job starts and failing cache operations while recovery completes. We will provide an update within the next 30 minutes..

Sep 29, 21:10:02 GMT+0
Investigating - Jobs across all regions are taking longer to start and cache operations are failing. We are continuing to investigate the root cause..

Duration: 25 minutes

Affected Components: eu-central Storage Cluster

Sep 29, 21:09:52 GMT+0
Completed - Due to the ongoing incident, the migration is rescheduled for another time, no change was applied..

Sep 29, 20:45:01 GMT+0
Identified - Maintenance is now in progress.

Sep 29, 20:42:46 GMT+0
Identified - We are migrating stickydisk metadata to a new database backend. No downtime expected..

Duration: 1 hour and 32 minutes

Affected Components: US West Cache

Sep 29, 18:32:38 GMT+0
Resolved - This incident is resolved. GitHub Actions cache operations for jobs in us-west have returned to normal, and any jobs that failed during the incident can be re-run..

Sep 29, 18:06:52 GMT+0
Monitoring - Cache errors for jobs in us-west have largely subsided. As we complete recovery, some jobs may see a one-time cache miss on their next run. We are monitoring and will provide an update within the next hour..

Sep 29, 17:35:30 GMT+0
Monitoring - We have restored the cache infrastructure in us-west, and cache failures have decreased but are not yet back to normal. Jobs in us-west may still see some cache restores and saves fail and fall back to a full install, while the earlier delayed job starts have cleared and other regions are unaffected. We are bringing additional cache capacity online to fully restore service..

Sep 29, 17:09:28 GMT+0
Identified - We have identified the cause of the failures, and we are implementing a fix to restore cache service. Customers with jobs in us-west may still see cache restores and saves fail and fall back to a full install, which can make those jobs run longer or fail, while jobs in other regions are not affected..

Sep 29, 17:00:21 GMT+0
Investigating - We are investigating failing GitHub Actions cache operations for jobs running in us-west since approximately 16:10 UTC. Affected jobs may see cache restores and saves fail and fall back to a full install, which can make those jobs run longer or fail, while jobs in other regions are not affected. .

Duration: 33 minutes

Affected Components: us-west x86, us-west ARM

Sep 24, 19:50:10 GMT+0
Resolved - Git checkout performance in us-west has remained stable since we rerouted traffic away from the congested upstream network link, and this incident is now resolved..

Sep 24, 19:31:46 GMT+0
Monitoring - After rerouting traffic, congestion on the upstream network link has subsided, and git checkout performance in us-west has returned to normal. We are monitoring and will resolve this incident once performance has continued to stay stable. .

Sep 24, 19:16:56 GMT+0
Investigating - We are investigating degraded network connectivity between our us-west region and GitHub. Customers running jobs in us-west may see slower-than-normal git checkouts, while jobs in other regions are not affected. Workflows using the Blacksmith checkout action with git checkout caching are less affected, and setup instructions are in the documentation below. Git checkout caching documentation: <https://docs.blacksmith.sh/blacksmith-caching/git-checkout-caching>.

Duration: 9 minutes

Affected Components: us-east x86, US East Cache, eu-west Storage Cluster, eu-central x86, eu-west ARM, us-west Storage Cluster, EU Central Cache, us-west x86, eu-west x86, EU West Cache, eu-west Storage Cluster, eu-west Storage Cluster, API, eu-central Storage Cluster, us-west ARM, EU West Runtime Build Cache, eu-central Storage Cluster, eu-central Storage Cluster, EU Central Runtime Build Cache, US West Runtime Build Cache, US West Cache, us-central MacOS, eu-central ARM, us-west Storage Cluster, us-west Storage Cluster

Sep 21, 18:47:13 GMT+0
Resolved - This issue has been resolved, and all services have operating normally. If any of your jobs failed during this time, please re-run them..

Sep 21, 18:37:44 GMT+0
Identified - We are seeing elevated errors in job adoption and other control plane APIs. This is yielding higher latencies for your job to be adopted. You may also see certain actions fail, such as actions caching and sticky disks. We have identified the issue and are rolling out a mitigation..

Duration: 47 minutes

Affected Components: us-east x86

Sep 13, 01:58:30 GMT+0
Resolved - This incident has been resolved..

Sep 13, 01:40:50 GMT+0
Monitoring - We have identified a small subset of runners that are exhibiting this network stalling and have isolated them. We're continuing to monitor recovery..

Sep 13, 01:11:55 GMT+0
Investigating - We are currently observing network stalling in our US East region with runner connectivity against AWS's us-east-1 region. Uploads and downloads to and from ECR and S3 may experience occasional stalling. We are investigating..

Duration: 9 hours and 30 minutes

Sep 11, 14:59:46 GMT+0
Resolved - This incident has been resolved - Canonical's Ubuntu apt mirrors have recovered and apt installs are succeeding again. We are also rolling out an image change to reduce the impact of upstream mirror outages in future..

Sep 11, 14:21:02 GMT+0
Monitoring - We are deploying a mitigation on our side that routes apt around the affected Canonical mirrors. For the time being pointing apt at [azure.archive.ubuntu.com](http://azure.archive.ubuntu.com) in place of [archive.ubuntu.com](http://archive.ubuntu.com) and [security.ubuntu.com](http://security.ubuntu.com) will unblock affected jobs..

Sep 11, 05:30:00 GMT+0
Monitoring - Since 06:30 UTC Canonical has had three outages affecting [archive.ubuntu.com](http://archive.ubuntu.com), [us.archive.ubuntu.com](http://us.archive.ubuntu.com) and [security.ubuntu.com](http://security.ubuntu.com). Although they have marked these as resolved, we are still seeing intermittent connection failures to those mirrors, which can cause apt-get steps to fail or hang. In the meantime, pointing apt at [azure.archive.ubuntu.com](http://azure.archive.ubuntu.com) instead of [archive.ubuntu.com](http://archive.ubuntu.com) and [security.ubuntu.com](http://security.ubuntu.com) will unblock affected jobs..

Duration: 1 hour and 20 minutes

Affected Components: eu-west Storage Cluster, us-west Storage Cluster, EU Central Cache, EU West Cache, eu-west Storage Cluster, eu-west Storage Cluster, eu-central Storage Cluster, US East Cache, EU West Runtime Build Cache, Dashboard, eu-central Storage Cluster, eu-central Storage Cluster, EU Central Runtime Build Cache, US West Runtime Build Cache, US West Cache, us-west Storage Cluster, us-west Storage Cluster

Sep 9, 18:06:25 GMT+0
Resolved - This incident has been resolved..

Sep 9, 17:21:48 GMT+0
Monitoring - We are seeing recovery of our caching infrastructure and our dashboard. We will continue to monitor this situation closely..

Sep 9, 16:46:00 GMT+0
Investigating - We are currently experiencing a major outage with both our caching infrastructure and our dashboards. These components are affected across all regions. We are investigating the issue and will provide an update soon..

Duration: 1 hour and 44 minutes

Affected Components: Github → API Requests

Sep 8, 16:44:09 GMT+0
Resolved - This incident is resolved. Job failures in our eu-west region returned to normal levels at 15:15 UTC and have remained there since; any jobs that failed between approximately 12:20 and 15:15 UTC with the error "The self-hosted runner lost communication with the server" can be safely re-run..

Sep 8, 15:31:33 GMT+0
Monitoring - Between approximately 12:20 and 15:15 UTC, some jobs in our eu-west region failed with the GitHub error "The self-hosted runner lost communication with the server"; other regions were unaffected. Job failures in eu-west returned to normal levels as of 15:15 UTC, and any affected jobs can be safely re-run while we continue to investigate the underlying cause..

Sep 8, 15:00:33 GMT+0
Investigating - Since approximately 12:20 UTC, some jobs in our eu-west region have been failing with the GitHub error "The self-hosted runner lost communication with the server"; other regions are unaffected. Affected jobs can be safely re-run while we continue to investigate the underlying cause. .

Duration: 3 hours and 10 minutes

Affected Components: us-west Storage Cluster, us-west Storage Cluster, us-west Storage Cluster

Sep 2, 19:23:31 GMT+0
Resolved - This incident is resolved. Sticky disk storage in our us-west region came under more read load than it could serve at normal latency, which slowed and produced higher error rates for Git-cached checkouts, incremental Docker builds, and container caching. We reduced and redistributed that load, and performance has been normal since approximately 17:15 UTC. Jobs that failed during the incident can be safely re-run. .

Sep 2, 18:10:23 GMT+0
Monitoring - Our us-west region storage cluster's latency and error rates have returned to baseline as of approximately 17:15 UTC, following mitigations that reduce load on the affected storage. Git-cached checkouts, incremental Docker builds, and container caching in us-west are back to normal, and we are monitoring to confirm the recovery holds while we continue to investigate the underlying cause. We will provide a final update within the next hour..

Sep 2, 17:22:52 GMT+0
Investigating - We are still working to resolve elevated latency on sticky disk storage in our us-west region; other regions are not affected. Customers running jobs in us-west may continue to see slower Git-cached checkouts, incremental Docker builds, and container caching. Our investigation into the underlying cause is ongoing and we will provide another update within the next 30 minutes. .

Sep 2, 16:48:35 GMT+0
Investigating - Sticky disk storage in our us-west region is experiencing higher latency; our other regions are not affected. Customers running jobs in us-west may see slower Git-cached checkouts, incremental Docker builds, and container caching. We are continuing to investigate the underlying cause and will provide another update within the next 30 minutes..

Sep 2, 16:13:47 GMT+0
Investigating - We are investigating reports of higher latency on sticky disk operations in our us-west region. Customers running jobs in us-west may see slower incremental Docker builds, Git caching, and container caching, so affected jobs can take longer than usual to complete. .

Duration: 2 hours and 22 minutes

Affected Components: Github → API Requests

Sep 2, 16:30:55 GMT+0
Resolved - This incident has been resolved. GitHub connectivity from our EU regions was degraded from \~12:45 to 16:15 UTC, affecting checkouts, [ghcr.io](http://ghcr.io), and some eu-west job starts. Normal since 16:15\. Affected jobs can be re-run..

Sep 2, 15:43:34 GMT+0
Monitoring - Connections to [github.com](http://github.com) and [ghcr.io](http://ghcr.io) from our eu-west, eu-central, and us-east regions have returned to normal as of approximately 15:00 UTC, so checkouts are completing at normal speed and login failures to GitHub Container Registry have dropped to baseline levels. We are continuing to monitor, and re-running any jobs that failed during this period should succeed..

Sep 2, 14:20:19 GMT+0
Investigating - We are seeing evidence that Git Checkouts are also affected by this. We are seeing high TCP retransmit rates into the EU GitHub loadbalancer. Checkouts in the EU West, EU Central, US East regions are affected..

Sep 2, 14:09:09 GMT+0
Investigating - We are currently observing higher rates of errors for the upstream GHCR registries which may indicate an undeclared GitHub incident. We are currently monitoring..

Duration: 2 hours and 29 minutes

Aug 18, 23:14:25 GMT+0
Resolved - This incident has been resolved..

Aug 18, 20:45:03 GMT+0
Monitoring - Jobs failure rate at an elevated rate caused by an upstream GitHub being rejected by 429 rate-limit errors. We are seeing a single-digit percentage increase in job failure rate across all organizations and are seeing similar failures on non-Blacksmith infrastructure..

Duration: 2 minutes

Affected Components: us-east x86, eu-central x86, eu-west ARM, us-west x86, eu-west x86, us-west ARM, us-central MacOS, eu-central ARM

Aug 17, 19:00:00 GMT+0
Resolved - This incident has been resolved..

Aug 17, 18:58:04 GMT+0
Investigating - We are seeing an increased error rate processing webhook events which will lead to delays in adoption jobs. We are actively investigating..

Duration: 6 hours and 57 minutes

Affected Components: Github → API Requests, Dashboard, Github → Webhooks

Aug 17, 20:40:52 GMT+0
Resolved - This incident has been resolved..

Aug 17, 19:32:23 GMT+0
Monitoring - We are no longer seeing upstream errors, and are continuing to monitor impact of the upstream outage..

Aug 17, 17:52:36 GMT+0
Monitoring - We are still seeing intermitting GitHub API errors at a low rate. .

Aug 17, 16:56:43 GMT+0
Monitoring - We're seeing signs of GitHub recovery. The dashboard is now loading and jobs should be running again. We are monitoring the recovery..

Aug 17, 13:46:46 GMT+0
Identified - Upstream incident has been declared: <https://www.githubstatus.com/incidents/zkxwbgr0cnmx> We are also seeing elevated error rates in jobs as they hit upstream GitHub errors. Job adoption times are also affected and are delayed. We are monitoring and are looking at potential mitigations..

Aug 17, 13:43:37 GMT+0
Identified - The Blacksmith Dashboard is unable to load. We've identified the root cause to upstream 503s being returned from GitHub on permission-check requests..

Duration: 4 hours

Affected Components: us-west Storage Cluster, us-west Storage Cluster, us-west Storage Cluster

Aug 16, 16:00:00 GMT+0
Completed - Maintenance has completed successfully.

Aug 16, 12:00:01 GMT+0
Identified - Maintenance is now in progress.

Aug 16, 11:47:45 GMT+0
Identified - We are planning for a scheduled maintenance during that time..

Duration: 1 day, 6 hours and 33 minutes

Affected Components: us-east x86, eu-central x86, eu-west ARM, us-west Storage Cluster, EU Central Cache, us-west x86, eu-west x86, EU West Cache, us-west ARM, US East Cache, US West Cache, eu-central ARM, Github → Webhooks, us-west Storage Cluster, us-west Storage Cluster

Aug 14, 18:14:17 GMT+0
Resolved - All services have been fully restored, including sticky disks, Docker container caching, and incremental Docker builders in us-west, and job queues are operating normally in all regions. A small amount of recently written cache data could not be recovered during storage repair, so some builds may run slower on their first runs while caches rebuild. This incident is now resolved, and we will publish a detailed post-incident report in the coming days..

Aug 14, 16:16:38 GMT+0
Monitoring - All services have been restored, including sticky disks, Docker container caching, and incremental Docker builders in us-west, and job queues are operating normally in all regions. We are continuing to monitor the stability of the recovered storage cluster as it ramps back up with traffic, and some builds may run slower on their first runs while recently written cache data rebuilds. We will post a final update once we have confirmed stability..

Aug 14, 15:02:08 GMT+0
Monitoring - The storage cluster for Sticky Disks, Docker Container Caching, and Incremental Docker Builders in us-west has been restored, and disks are mounting and operating normally. As part of the recovery, a small amount of recently written cache data may need to be rebuilt, so some Docker builds may run slower over their first few runs while caches rehydrate. We are monitoring closely..

Aug 14, 10:57:11 GMT+0
Monitoring - We are continuing to work with our compute provider to restore the storage cluster for Sticky Disks, Docker Container Caching, and Incremental Docker Builders in us-west. This requires hands-on recovery work by our provider's team and may take a few more hours to fully restore. Workflows using the affected features in us-west may see slow or stalled Docker builds in the meantime..

Aug 14, 06:06:54 GMT+0
Monitoring - We have restored the Github Actions Cache in us-west. We are still working with our compute provider to restore our remaining storage cluster for Sticky Disks, Docker Container Caching, and Incremental Docker Builders..

Aug 14, 04:09:30 GMT+0
Monitoring - We are still working with our compute provider to recover the storage clusters..

Aug 14, 03:26:24 GMT+0
Monitoring - We are noticing some issues with the storage cluster backing the Github Actions cache after it was restored. We are working on restoring this alongside the storage cluster backing Sticky Disks, Incremental Docker Builders, and Docker Container Caching..

Aug 14, 03:06:21 GMT+0
Monitoring - Our us-west compute provider has restored power and cooling in their datacenter and the majority of our capacity has returned. Github Actions caching has been reenabled in the region but we are still working to restore Sticky Disks, Docker Container caching, and Incremental Docker Builders. We will post an update once these components are restored..

Aug 14, 02:08:56 GMT+0
Monitoring - Job queues have fully recovered, and runners in all regions are operating normally for all runner sizes. Cache operations, including sticky disks, remain unavailable in us-west while we bring the restored storage hardware back online. We will provide another update within the next hour..

Aug 14, 01:15:24 GMT+0
Monitoring - Queue times for all runner sizes have returned to normal in all regions, and we are monitoring closely while eu-west clears the last of its backlog. Cache operations, including sticky disks, remain unavailable in us-west while we bring the restored storage hardware back online. We will provide another update within the next hour..

Aug 14, 00:00:40 GMT+0
Identified - Runner capacity in us-west continues to recover and queues in all regions are draining. Our provider has restored the access we need to begin bringing our us-west storage systems back online. Cache operations, including sticky disks, remain unavailable while that work completes, and jobs may still take longer than usual to start..

Aug 13, 22:54:06 GMT+0
Identified - Queue times are improving across all regions as capacity comes back online, but jobs may still take longer than usual to start. Cache operations in us-west, including sticky disks, remain unavailable while our provider restores the storage systems, and jobs on our largest runner sizes may see the longest delays. We will provide another update within the next hour..

Aug 13, 21:48:02 GMT+0
Identified - Queue times in us-west have returned to near-normal levels, while us-east, eu-west, and eu-central are still working through their remaining backlogs. Cache operations, including sticky disks, remain unavailable in us-west while our provider works to restore the storage systems there. We will provide another update within the next hour..

Aug 13, 20:44:17 GMT+0
Identified - Queues in us-west have come down substantially as restored capacity comes online and we continue moving customers back. Cache operations, including sticky disks, remain unavailable in us-west, and jobs in us-east may still take significantly longer than usual to start while we work through the remaining backlog. We will provide another update within the next hour..

Aug 13, 19:59:27 GMT+0
Identified - More us-west capacity has come back online and queues in the region are steadily draining as we move customers back. Cache operations, including sticky disks, remain unavailable in us-west while our provider works to restore the storage portion of the facility, and because we shifted us-west traffic to other regions earlier today, jobs elsewhere, particularly in us-east, may still take longer than usual to start. We will provide another update within the next hour..

Aug 13, 19:15:14 GMT+0
Identified - Our provider is continuing to bring us-west servers back online, and we are gradually moving customers back to us-west as capacity returns. The backlog has not yet cleared, so jobs in all regions may still take longer than usual to start. We will provide another update within the next hour..

Aug 13, 18:26:30 GMT+0
Identified - Our provider has begun powering servers back on at the us-west facility, and a portion of our us-west capacity is back online. We have started moving some customers back to us-west to spread load across regions. Jobs in all regions may still take longer than usual to start..

Aug 13, 17:42:15 GMT+0
Identified - Our provider is continuing to restore cooling at the us-west facility; temperatures have not yet reached safe levels for servers to be powered back on, and we are staged to begin restoring runner capacity as soon as they are. The us-west region remains in a major outage, and jobs in all regions may still take longer than usual to start. We will provide another update within the next hour..

Aug 13, 17:00:03 GMT+0
Identified - Our provider has partially restored cooling at the us-west facility and expects to reach safe operating temperatures within the next few hours, at which point servers will be powered back on in phases. In the meantime jobs in all regions may still take significantly longer than usual to start. We will provide another update within the next hour..

Aug 13, 16:32:39 GMT+0
Identified - Our datacenter provider has begun deploying mitigations at the affected us-west facility, and GitHub has resolved its separate webhooks incident, though some knock-on delays may persist while affected jobs requeue. The us-west region remains heavily affected and with capacity reduced while we rebalance traffic, jobs in all regions may take longer than usual to start..

Aug 13, 15:11:27 GMT+0
Identified - Conditions in our us-west region have not yet improved. We're actively working with our datacenter provider on the issue, and will provide updates as they occur. We are also manually rebalancing traffic out of the us-west region to aid with recovery. Github have also declared an incident affecting webhooks - <https://www.githubstatus.com/incidents/k8vbzwqjkxzn> which may cause some job adoption delays.

Aug 13, 14:40:13 GMT+0
Identified - Conditions at our provider's us-west facility have not yet improved. We are continuing to issue mitigations to ensure that customer impact is minimal. .

Aug 13, 13:59:44 GMT+0
Identified - Conditions at our provider's us-west facility have not yet improved, and we are continuing to work around the issue to keep customer impact minimal. Such mitigations may include rebalancing customers to unaffected regions..

Aug 13, 13:27:12 GMT+0
Identified - Conditions at our provider's us-west facility have not yet improved. As we rebalance customers away from us-west, other regions may see slightly longer pickup times than usual as a side effect.

Aug 13, 13:02:39 GMT+0
Identified - Conditions at our provider's us-west facility have not yet improved, and we are continuing to work around the issue to keep customer impact minimal. Such mitigations may include rebalancing customers to unaffected regions..

Aug 13, 12:36:49 GMT+0
Identified - We are currently applying mitigations for affected customers in the region..

Aug 13, 12:06:56 GMT+0
Identified - Caching performance in US West is currently degraded. Our upstream provider has lost cooling power in their data center, leading to failure of some storage systems in US West. As a result jobs may take longer to complete. We are taking action to mitigate the issue..

Aug 13, 11:40:58 GMT+0
Investigating - Our cloud provider is experiencing a thermal event in their us-west datacenter that is affecting caching in the region. Sticky Disk, Actions Cache, Docker Container Cache, and Bazel Build Caching are impacted: cache operations may be slow or unavailable which may result in some builds running slower than usual..

Duration: 6 hours and 39 minutes

Affected Components: Github → Actions

Aug 13, 01:20:46 GMT+0
Resolved - This incident has been resolved..

Aug 12, 23:06:43 GMT+0
Monitoring - GitHub has implemented a fix and we are beginning to see recovery. We will continue to monitor the issue. <https://www.githubstatus.com/incidents/lsvy8xsf0gxv>.

Aug 12, 22:03:11 GMT+0
Identified - GitHub has confirmed an incident affecting release asset downloads, consistent with our findings (githubstatus.com/incidents/lsvy8xsf0gxv). Workflows downloading GitHub release assets remain affected across all regions while GitHub works toward mitigation, and retrying failed jobs remains the most effective workaround. .

Aug 12, 21:31:36 GMT+0
Identified - Workflows downloading GitHub release assets remain affected across all regions because of an upstream issue at GitHub, and retrying failed jobs remains the most effective workaround..

Aug 12, 20:52:19 GMT+0
Identified - Workflows downloading GitHub release assets remain affected across all regions, and retrying failed jobs remains the most effective workaround. The failures remain isolated to a subset of GitHub's edge servers upstream of our infrastructure, and we are continuing to monitor for recovery..

Aug 12, 20:23:55 GMT+0
Identified - We have isolated the cause to a subset of GitHub's edge servers and verified the failures occur from networks outside Blacksmith as well. Workflows downloading GitHub release assets remain affected across all regions, and retrying failed jobs is the most effective workaround until the upstream issue is resolved. We will provide another update within the next 30 minutes..

Aug 12, 19:55:41 GMT+0
Identified - We are still experiencing an elevated rate of socket hangup failures when workflows download GitHub release assets, across all regions. The failures remain isolated to GitHub upstream, and we are continuing to monitor for recovery; other GitHub interactions, including repository checkout, remain unaffected. We will provide another update within the next 30 minutes..

Aug 12, 19:25:16 GMT+0
Identified - We are still experiencing an elevated rate of socket hangup failures when workflows download GitHub release assets, across all regions. The failures have been isolated to GitHub upstream, and we are monitoring closely while the issue persists. We have observed multiple public reports that GitHub is experiencing a partial outage..

Aug 12, 18:41:51 GMT+0
Investigating - We are investigating reports of socket hangups when attempting to fetch GitHub release assets. There are not any current upstream incidents, but we are continuing to investigate..

Duration: 8 hours and 21 minutes

Affected Components: us-west x86

Aug 12, 11:50:47 GMT+0
Resolved - This incident has been resolved..

Aug 12, 11:26:10 GMT+0
Monitoring - We implemented a fix and are seeing improvements. We are continuing to monitor the result..

Aug 12, 10:33:01 GMT+0
Identified - We are currently deploying a mitigation.

Aug 12, 09:10:10 GMT+0
Identified - We have identified the affected mirror and are implementing a fix.

Aug 12, 03:30:00 GMT+0
Investigating - We are currently investigating this incident..

Duration: 1 hour

Affected Components: , Actions Cache →

Aug 11, 20:00:00 GMT+0
Resolved - We resolved the configuration error leading to the increased rate of cache misses. Cache hit rates are now at the baseline..

Aug 11, 19:00:10 GMT+0
Investigating - We are observing certain customers experiencing higher than baseline cache miss rates..

Duration: 1 hour and 27 minutes

Affected Components: us-west x86, us-west ARM, Dashboard

Aug 10, 20:44:19 GMT+0
Resolved - This incident is resolved, with job performance and the ingestion of metrics and logs in our us-west region stable for the past 30 minutes. Jobs that failed or timed out during the incident can be safely re-run. .

Aug 10, 20:18:51 GMT+0
Monitoring - Metrics and log ingestion in our us-west region is recovering, and job performance in the region has returned to normal. We are monitoring to confirm the recovery holds and are continuing to investigate the underlying cause. We will provide an update shortly..

Aug 10, 19:45:37 GMT+0
Investigating - We are experiencing degraded network performance in our us-west region, affecting metrics and log ingestion as well as job performance. Jobs in us-west that upload artifacts or transfer large amounts of data may run slower than normal and in some cases hit their configured timeouts and fail. Other regions are not affected, and we are actively investigating the issue..

Aug 10, 19:44:03 GMT+0
Investigating - We are continuing to investigate degraded metrics and log ingestion in our us-west region. Customers may still see metrics and logs for their jobs appear missing or delayed in the Blacksmith dashboard, while other regions remain unaffected. We will provide another update within the next 30 minutes..

Aug 10, 19:17:34 GMT+0
Investigating - We are experiencing degradation in our metrics and log ingestion services in our us-west region. We are actively investigating the issue..

Duration: 20 days, 14 hours and 54 minutes

Affected Components: us-west Storage Cluster, eu-central Storage Cluster

Aug 8, 20:00:00 GMT+0
Completed - Maintenance has completed successfully.

Aug 8, 12:00:01 GMT+0
Identified - Maintenance is now in progress.

Aug 8, 11:45:06 GMT+0
Identified - We are increasing Ceph background ops activity, elevating the overall load. No downtime expected..

Duration: 9 hours and 27 minutes

Affected Components: Github → Webhooks, Github → Actions

Aug 7, 00:57:44 GMT+0
Resolved - GitHub job success and adoption rates are now back at normal levels. Jobs that were not adopted during the incident are being requeued by our team and should be picked up shortly..

Aug 6, 15:30:48 GMT+0
Monitoring - GitHub has reported degraded performance for Actions. Jobs may be delayed to start, or fail during the set up step. We are monitoring this incident. <https://www.githubstatus.com/incidents/qcvjkzcs7j74>.

Duration: 1 hour and 34 minutes

Affected Components: eu-west Storage Cluster, eu-central x86, eu-west ARM, us-west x86, us-west Storage Cluster, , eu-west x86, eu-west Storage Cluster, eu-west Storage Cluster, , us-west ARM, eu-central Storage Cluster, , Codesmith, eu-central Storage Cluster, eu-central Storage Cluster, https://blacksmith.sh, us-central MacOS, eu-central ARM, us-west Storage Cluster, us-west Storage Cluster, Runtime Build Caching → Actions Cache → Website →

Aug 6, 02:53:11 GMT+0
Resolved - This incident has been resolved..

Aug 6, 02:45:02 GMT+0
Monitoring - Job adoption is now fully operational..

Aug 6, 02:11:10 GMT+0
Monitoring - There is still a delay in job adoption times as we continue to recover..

Aug 6, 01:51:15 GMT+0
Monitoring - Jobs adoption has recovered and are operational..

Aug 6, 01:36:40 GMT+0
Identified - We've mitigated the root cause and jobs are resuming to run. Some jobs may still be delayed to start as we catch up with the job backlog..

Aug 6, 01:18:55 GMT+0
Investigating - We're currently experiencing an outage of our control plane. GitHub job adoption and execution are affected, as well as dashboard access..

Duration: 53 minutes

Affected Components: eu-west Storage Cluster, us-west Storage Cluster, , eu-west Storage Cluster, eu-west Storage Cluster, , eu-central Storage Cluster, eu-central Storage Cluster, eu-central Storage Cluster, us-west Storage Cluster, us-west Storage Cluster, Runtime Build Caching → Actions Cache →

Aug 5, 23:37:56 GMT+0
Resolved - This incident has been resolved..

Aug 5, 23:30:42 GMT+0
Monitoring - We are still monitoring recovery..

Aug 5, 22:55:00 GMT+0
Monitoring - We implemented a fix and are currently monitoring the result..

Aug 5, 22:45:00 GMT+0
Investigating - We are seeing elevated error rates for Sticky Disk and Actions Cache requests. We are investigating..

Duration: 6 minutes

Affected Components: eu-central x86, eu-west ARM, us-west x86, eu-west x86, us-west ARM, us-central MacOS, eu-central ARM

Aug 3, 15:42:46 GMT+0
Resolved - This incident has been resolved..

Aug 3, 15:37:05 GMT+0
Investigating - We're seeing a number of cases where jobs may not be adopted promptly. We're currently investigating. .

Duration: 21 minutes

Affected Components: us-west Storage Cluster, us-west Storage Cluster, us-west Storage Cluster

Aug 2, 06:37:06 GMT+0
Resolved - This incident has been resolved..

Aug 2, 06:25:32 GMT+0
Monitoring - We've applied a fix and are seeing failure rates and latency starting to come down..

Aug 2, 06:16:09 GMT+0
Investigating - We are seeing some failures with sticky disk availability and increased latency with the US-West storage cluster..

Duration: 1 hour and 44 minutes

Affected Components: , Runtime Build Caching →

Jul 30, 03:13:57 GMT+0
Completed - Maintenance has completed successfully. Runtime Build Caching has been fully restored and is operating normally..

Jul 30, 01:30:01 GMT+0
Identified - Maintenance is now in progress.

Jul 30, 01:15:54 GMT+0
Identified - We will be performing maintenance on our Runtime Build Caching infrastructure. Jobs will continue to run, but some may run without remote caching and take longer than usual during this window. No customer action is required. We expect maintenance to last approximately two hours..

Duration: 26 minutes

Affected Components: Codesmith

Jul 29, 16:31:17 GMT+0
Resolved - This incident has been resolved..

Jul 29, 16:08:02 GMT+0
Monitoring - The upstream issue is resolved, we are monitoring the sandbox creation rate recovering..

Jul 29, 16:05:10 GMT+0
Identified - We're seeing increased error rates from an upstream provider in spinning up sandbox environments in Codesmith. We are looking into remediation options..

Duration: 34 minutes

Affected Components: eu-central x86, us-west x86, eu-west ARM, eu-west x86, us-west ARM, us-central MacOS, eu-central ARM

Jul 24, 19:51:29 GMT+0
Resolved - This incident has been resolved, and job pickup has returned to normal..

Jul 24, 19:23:36 GMT+0
Monitoring - We have requeued the webhooks that were not processed during the incident, and any affected jobs should start shortly..

Jul 24, 19:20:01 GMT+0
Monitoring - We have deployed a fix and are monitoring recovery, and will provide another update within the next 30 minutes..

Jul 24, 19:17:34 GMT+0
Investigating - We are seeing webhooks not being accepted by our control plane resulting in jobs not being adopted. We are currently investigating..

Duration: 4 hours and 50 minutes

Affected Components: us-west Storage Cluster

Jul 13, 00:50:27 GMT+0
Completed - Maintenance has completed successfully..

Jul 12, 20:00:01 GMT+0
Identified - Maintenance is now in progress.

Jul 12, 15:05:49 GMT+0
Identified - We will be performing Ceph major version upgrade. No downtime expected..

Duration: 2 hours and 34 minutes

Affected Components: eu-west Storage Cluster, eu-central Storage Cluster

Jul 12, 15:03:54 GMT+0
Completed - Maintenance has completed successfully..

Jul 12, 12:30:01 GMT+0
Identified - Maintenance is now in progress.

Jul 12, 12:25:50 GMT+0
Identified - We will be performing Ceph major version upgrade. No downtime expected..

Duration: 1 hour

Affected Components: , Actions Cache →

Jul 3, 01:00:00 GMT+0
Completed - Maintenance has completed successfully.

Jul 3, 00:00:01 GMT+0
Identified - Maintenance is now in progress.

Jul 2, 23:58:28 GMT+0
Identified - We are planning for a scheduled maintenance on our Actions Cache storage cluster. This is to address a kernel bug that we have discovered resulting in sporadic instability on the machines hosted this cluster. Customers can expect to see blips in their Actions Cache reads and writes during this period but they should not be fatal to the job. At the end of the maintenance all cached data will be reset so your jobs may run slower as they hydrate this cache..

Duration: 1 hour

Affected Components: , Actions Cache →

Jul 2, 01:00:00 GMT+0
Completed - Maintenance has completed successfully..

Jul 2, 00:30:00 GMT+0
Identified - We are planning for a scheduled maintenance during that time. Cache states will be reset upon completion..

Affected Components: eu-central Storage Cluster, eu-central Storage Cluster, eu-central Storage Cluster

Jun 11, 15:59:04 GMT+0
Completed - Maintenance has completed successfully..

Jun 11, 15:58:54 GMT+0
Identified - We are planning for a scheduled maintenance during that time..

Duration: 1 hour and 15 minutes

Affected Components: us-west Storage Cluster, us-west Storage Cluster, us-west Storage Cluster

Jun 11, 04:10:37 GMT+0
Completed - Maintenance has completed successfully..

Jun 11, 02:55:19 GMT+0
Identified - Maintenance is now in progress..

Jun 11, 02:55:02 GMT+0
Identified - We are planning for a scheduled maintenance during that time..

Duration: 1 hour

Affected Components: us-west Storage Cluster, us-west Storage Cluster, us-west Storage Cluster

Jun 10, 02:00:00 GMT+0
Completed - Maintenance has completed successfully.

Jun 10, 01:00:01 GMT+0
Identified - Maintenance is now in progress.

Jun 10, 01:00:00 GMT+0
Identified - We're seeing some service degradation post maintenance so we need to extend the maintenance window..

Duration: 20 minutes

Affected Components: us-west Storage Cluster, us-west Storage Cluster, us-west Storage Cluster

Jun 9, 23:50:10 GMT+0
Completed - Maintenance has completed successfully..

Jun 9, 23:30:01 GMT+0
Identified - Maintenance is now in progress.

Jun 9, 23:30:00 GMT+0
Identified - We're planning an emergency maintenance for the US Storage Cluster..

Duration: 1 hour and 19 minutes

Affected Components: eu-central Storage Cluster, eu-central Storage Cluster, eu-central Storage Cluster

May 1, 21:49:24 GMT+0
Completed - Maintenance has completed successfully..

May 1, 20:30:00 GMT+0
Identified - We are planning for a emergency maintenance at this time. During this maintenance the Storage Cluster components will have degraded performance. Existing caches will be wiped upon maintenance completion..

May 1, 20:30:00 GMT+0
Identified - We are planning for a emergency maintenance at this time. During this maintenance the Storage Cluster components will have degraded performance. Existing caches will be wiped upon maintenance completion..

Duration: 1 hour

Affected Components: , , , Sticky Disks → Docker Container Cache → Incremental Docker Builders →

Apr 18, 17:00:00 GMT+0
Completed - Maintenance has completed successfully.

Apr 18, 16:00:01 GMT+0
Identified - Maintenance is now in progress.

Apr 18, 16:00:00 GMT+0
Identified - We are planning for a scheduled maintenance during that time..

Duration: 1 hour

Affected Components: , Incremental Docker Builders →

Apr 12, 05:00:00 GMT+0
Completed - Maintenance has completed successfully.

Apr 12, 04:00:01 GMT+0
Identified - Maintenance is now in progress.

Apr 12, 04:00:00 GMT+0
Identified - We are planning for a scheduled maintenance during that time..

Duration: 20 minutes

Affected Components: , Incremental Docker Builders →

Apr 8, 18:55:38 GMT+0
Completed - Maintenance has completed successfully..

Apr 8, 18:45:00 GMT+0
Identified - Maintenance is now in progress..

Apr 8, 18:35:00 GMT+0
Identified - We are planning for an emergency maintenance on our storage cluster to size it up to eliminate degraded interactions. All sticky disk cache entries will be wiped as part of this maintenance..

Duration: 15 minutes

Affected Components: , Actions Cache →

Mar 20, 01:15:00 GMT+0
Completed - Maintenance has completed successfully.

Mar 20, 01:00:01 GMT+0
Identified - Maintenance is now in progress.

Mar 20, 01:00:00 GMT+0
Identified - We are upgrading the Actions Cache capacity, cache performance may be degraded during this maintenance window..

Duration: 4 minutes

Affected Components: us-west ARM, eu-central x86, us-west x86, eu-central ARM, , Blacksmith Managed Runners →

Feb 28, 18:10:10 GMT+0
Completed - Maintenance has completed successfully..

Feb 28, 18:07:00 GMT+0
Identified - We are running an operation on our database that may result in a short blip in service. Customers may notice longer queue times during this window..

Feb 28, 18:05:51 GMT+0
Identified - Maintenance is now in progress..

Duration: 9 minutes

Affected Components: , Incremental Docker Builders →

Feb 26, 00:37:41 GMT+0
Completed - Maintenance has completed successfully the storage cluster is back to serving requests normally..

Feb 26, 00:28:40 GMT+0
Identified - Maintenance is now in progress..

Feb 25, 14:00:00 GMT+0
Identified - We are planning for an emergency maintenance on our storage cluster to size it up to eliminate degraded interactions. All sticky disk cache entries will be wiped as part of this maintenance..

Duration: 1 hour

Affected Components: , Incremental Docker Builders →

Aug 6, 19:30:00 GMT+0
Completed - Maintenance has completed successfully..

Aug 6, 18:30:00 GMT+0
Identified - We are going to be running an important upgrade to our storage fleet cluster that may cause interruptions in stickydisk and docker caching. The upgrade will cause a reset of all stickydisk entities causing runs to be uncached for the first time until the cache is rehydrated..