Status | Blacksmith - Notice history

We've identified the root cause to be a transient issue with GitHub's control plane that lasted for ~15 minutes. GitHub’s Actions broker service began rejecting session creation requests from our runners. While our runners were provisioned successfully, they were being rejected by the GitHub Actions broker service, causing the runner to exit without picking up a job.

This led to job adoption observed delays. The issue resolved on its own as GitHub’s broker service recovered. We also manually requeued jobs that were affected.

We're seeing recovery in job adoption and queue times are no longer elevated. We will continue to monitor.

We've implemented a fix and are seeing complete recovery. We will continue to monitor.

This incident has been resolved.

We're noticing recovery across the US fleet for GitHub interactions since the traffic was moved off the identified carrier.

Our data center provider has identified packet loss with a specific network carrier and are in the process of moving traffic off them

The incident has now been resolved.

We were seeing severe tail latency during git checkout (minutes+) consistently correlated with a small set of GitHub edge IPs in the Ashburn region. Median checkout times were largely fine, but ~0.5–1% of operations hang for a very long time.

We ruled out network side issues on our end and worked with our vendors to rule out congestion or packet loss. All signs pointed to a degraded edge node rather than a client or network issue. We were also seeing this intermittently affect GitHub-hosted runners, but it was more prominent on Blacksmith runners given our checkout concurrency and tail-latency instrumentation.

We are actively working on solutions that will allow us to direct traffic to a different edge node in such situations in the future. We apologize for the inconvenience we may have caused.

We are continuing to investigate the issue.

Queue times from ARM runners in our US region have returned to normal.

Queue times from ARM runners in our US region have returned to normal.

We are continuing to monitor elevated queue times for ARM runners in our US region.

We'e continuing to monitor elevated queue times for ARM runners.

Queue times have returned to normal. These elevated queue times were due to a misconfigured configuration that has been rolled back. This incident has been resolved.

We are continuing to investigate elevated queue times for ARM runners in the US.