blacksmith - Notice history

We are continuing to see recovery in us-west

This incident has been resolved.

The re-queuing is still in progress however new jobs are being picked up without delay.

We have completed the DB operation and currently re-queuing jobs. This may take ~15 minutes

We needed to perform an emergency maintenance operation in our storage cluster. We are observing the recovery and seeing docker build speeds improve.

Queue times are no longer elevated. We are continuing to monitor.

We're seeing signs of recovery and are continuing to monitor.

We are continuing to investigate delays in job adoption and degraded Git interactions, mostly for Checkout actions.

This appears to be related to an upstream GitHub outage. We are monitoring the situation and will provide updates as more information becomes available.

We've identified the root cause to be a transient issue with GitHub's control plane that lasted for ~15 minutes. GitHub’s Actions broker service began rejecting session creation requests from our runners. While our runners were provisioned successfully, they were being rejected by the GitHub Actions broker service, causing the runner to exit without picking up a job.

This led to job adoption observed delays. The issue resolved on its own as GitHub’s broker service recovered. We also manually requeued jobs that were affected.

We're seeing recovery in job adoption and queue times are no longer elevated. We will continue to monitor.