The incident has now been resolved.
We were seeing severe tail latency during git checkout (minutes+) consistently correlated with a small set of GitHub edge IPs in the Ashburn region. Median checkout times were largely fine, but ~0.5–1% of operations hang for a very long time.
We ruled out network side issues on our end and worked with our vendors to rule out congestion or packet loss. All signs pointed to a degraded edge node rather than a client or network issue. We were also seeing this intermittently affect GitHub-hosted runners, but it was more prominent on Blacksmith runners given our checkout concurrency and tail-latency instrumentation.
We are actively working on solutions that will allow us to direct traffic to a different edge node in such situations in the future. We apologize for the inconvenience we may have caused.
