Status | Blacksmith - Notice history

Queue times are no longer elevated. We're continuing to monitor.

Queue times remain elevated for ARM runners in the US.

We're seeing signs of recovery and are continuing to monitor.

We are continuing to investigate delays in job adoption and degraded Git interactions, mostly for Checkout actions.

This appears to be related to an upstream GitHub outage. We are monitoring the situation and will provide updates as more information becomes available.

We've identified the root cause to be a transient issue with GitHub's control plane that lasted for ~15 minutes. GitHub’s Actions broker service began rejecting session creation requests from our runners. While our runners were provisioned successfully, they were being rejected by the GitHub Actions broker service, causing the runner to exit without picking up a job.

This led to job adoption observed delays. The issue resolved on its own as GitHub’s broker service recovered. We also manually requeued jobs that were affected.

We're seeing recovery in job adoption and queue times are no longer elevated. We will continue to monitor.

This incident has been resolved.

We're noticing recovery across the US fleet for GitHub interactions since the traffic was moved off the identified carrier.

Our data center provider has identified packet loss with a specific network carrier and are in the process of moving traffic off them

The incident has now been resolved.

We were seeing severe tail latency during git checkout (minutes+) consistently correlated with a small set of GitHub edge IPs in the Ashburn region. Median checkout times were largely fine, but ~0.5–1% of operations hang for a very long time.

We ruled out network side issues on our end and worked with our vendors to rule out congestion or packet loss. All signs pointed to a degraded edge node rather than a client or network issue. We were also seeing this intermittently affect GitHub-hosted runners, but it was more prominent on Blacksmith runners given our checkout concurrency and tail-latency instrumentation.

We are actively working on solutions that will allow us to direct traffic to a different edge node in such situations in the future. We apologize for the inconvenience we may have caused.

We are continuing to investigate the issue.