blacksmith - Notice history

The sticky disks cluster has been recovered, some sticky disks have been purged. Monitoring for any lingering stuckness.

This incident has been resolved.

This incident has been resolved. Download speeds in us-east and us-west are back to normal, and affected jobs can be re-run.

Download speeds in us-west have recovered and both US regions are back to operational. Affected jobs can be re-run and should complete normally. We are continuing to monitor and will post an update once we have confirmed stability.

Download speeds for jobs in us-east have recovered and the region is back to operational. Jobs in us-west are still seeing slow downloads from GitHub while our network provider works on the issue.

This incident is resolved. macOS capacity has been restored and macOS jobs have been picking up normally since this evening.

The macOS job queue has cleared and new 6vcpu and 12vcpu macOS jobs are picking up normally across all regions. We are continuing to restore the remaining capacity with our upstream provider and will keep monitoring before resolving this incident. Linux and Windows runners were unaffected throughout.

The queue for macOS jobs is subsiding as restored capacity continues to come online with our upstream provider, and new 6vcpu and 12vcpu macOS jobs are picking up normally for most customers. A small number of 12vcpu macOS jobs queued earlier today are still being worked through and may take longer to start, while Linux and Windows runners are unaffected. We are continuing to monitor and will resolve this incident once the remaining backlog has cleared.

New macOS jobs are starting within a few minutes for most customers, though some 6vcpu macOS jobs may wait up to 10 minutes during busy periods while we finish restoring capacity with our upstream provider. A backlog of 12vcpu macOS jobs queued earlier today is still being worked through, and those jobs may take longer to start, while Linux and Windows runners are unaffected.

6vcpu macOS jobs are now starting within a few minutes, and the backlog of 12vcpu macOS jobs is draining as we continue to add capacity with our upstream provider. Customers with 12vcpu macOS jobs already queued may still see longer waits for them to start, while new jobs are picking up faster, and Linux and Windows runners are unaffected

We have brought additional macOS capacity online and queue times have started to improve. Customers should currently expect 6vcpu macOS jobs to wait a few minutes to start and 12vcpu macOS jobs considerably longer, while Linux and Windows runners are unaffected.

We are continuing to bring macOS hosts back into service and are working with our upstream provider to release the remaining hosts. Customers should currently expect 6vcpu macOS jobs to wait around 10 minutes to start and 12vcpu macOS jobs considerably longer. Linux and Windows runners are unaffected.

We are continuing to bring macOS hosts back into service in batches and are working with our upstream provider to release the remaining hosts. Customers should currently expect 6vcpu macOS jobs to wait around 10 minutes to start and 12vcpu macOS jobs considerably longer, while Linux and Windows runners remain unaffected.

We have returned an additional group of macOS hosts to service and expect a further large batch to come online within the next 30 minutes, at which point waiting macOS jobs should start being picked up more quickly. Customers should currently expect 6vcpu macOS jobs to wait around 5 minutes to start, while 12vcpu macOS jobs are waiting considerably longer, and Linux and Windows runners are unaffected.

We are continuing to bring additional macOS hosts back online with our upstream provider, and the capacity restored earlier remains in service. Customers may still see longer than normal queue times for macOS jobs, with 12vcpu macOS jobs most affected, while Linux and Windows runners are unaffected.

We have restored a portion of the affected macOS capacity and are continuing to work with our upstream provider to bring the remaining hosts back online. Customers should still expect longer than normal queue times for macOS jobs, with 12vcpu macOS jobs most affected, while Linux and Windows runners are unaffected.

We are continuing to work with our upstream provider to bring the affected macOS hosts back online. Customers should still expect longer than normal queue times for macOS jobs. We will provide another update shortly.

We are continuing to bring additional macOS hosts online with our upstream provider however available macOS capacity has not yet recovered. Customers can still expect longer than normal queue times for macOS jobs, with 12vcpu macOS jobs most affected. We will provide another update shortly

This incident has been resolved.

We've rolled out a fix and are currently monitoring the results. We are seeing major recovery in the request latencies in our dashboard.

We've applied a few mitigations and are seeing some improvement. Certain queries are still timing out. We're still investigating.

This incident has been resolved.

This incident is resolved. Job start times and cache operations returned to normal in all regions, and we are continuing to monitor. Jobs that failed during the incident can be re-run.

Caching has recovered, and job queue times across EU regions have recovered. Queue times for larger jobs (16 and 32 vcpu jobs) in US west and US east continue to remain elevated. We are continuing to monitor for any regressions.

Our mitigation has taken effect and job start times and cache operations are improving. Customers may still see some delayed job starts and cache failures while recovery completes. We will provide an update within the next 30 minutes.

We have applied a fix and are monitoring its effect. Customers may still see delayed job starts and failing cache operations while recovery completes. We will provide an update within the next 30 minutes.

This incident is resolved. GitHub Actions cache operations for jobs in us-west have returned to normal, and any jobs that failed during the incident can be re-run.

Cache errors for jobs in us-west have largely subsided. As we complete recovery, some jobs may see a one-time cache miss on their next run. We are monitoring and will provide an update within the next hour.

We have restored the cache infrastructure in us-west, and cache failures have decreased but are not yet back to normal. Jobs in us-west may still see some cache restores and saves fail and fall back to a full install, while the earlier delayed job starts have cleared and other regions are unaffected. We are bringing additional cache capacity online to fully restore service.

We have identified the cause of the failures, and we are implementing a fix to restore cache service. Customers with jobs in us-west may still see cache restores and saves fail and fall back to a full install, which can make those jobs run longer or fail, while jobs in other regions are not affected.