blacksmith - Slow checkouts and downloads in us-east and us-west – Incident details

Slow checkouts and downloads in us-east and us-west

ResolvedDegraded performance1 hour 6 minutes

Affected

  • ResolvedUTC

    This incident has been resolved. Download speeds in us-east and us-west are back to normal, and affected jobs can be re-run.

  • MonitoringUTC

    Download speeds in us-west have recovered and both US regions are back to operational. Affected jobs can be re-run and should complete normally. We are continuing to monitor and will post an update once we have confirmed stability.

  • IdentifiedUTC

    Download speeds for jobs in us-east have recovered and the region is back to operational. Jobs in us-west are still seeing slow downloads from GitHub while our network provider works on the issue.

  • InvestigatingUTC

    Jobs on Linux runners in us-east are seeing severely slow downloads from GitHub and Google services and in us-west slow downloads from GitHub. This includes actions/checkout, git clone/git fetch, downloading actions from github.com, and pulls from Google Container Registry / Artifact Registry. Jobs that depend on these may run much longer than usual or time out.

    The job slowdown is also leading to increased job adoption times in those regions.