blacksmith - Delays in Job Adoption – Incident details

Delays in Job Adoption

Resolved19 minutes

Affected

Blacksmith Managed Runners

Degraded performance from 5:30 PM to 5:44 PM, Operational from 5:44 PM to 5:50 PM

US West

Degraded performance from 5:30 PM to 5:44 PM, Operational from 5:44 PM to 5:50 PM

us-west ARM

Degraded performance from 5:30 PM to 5:44 PM, Operational from 5:44 PM to 5:50 PM

us-west x86

Degraded performance from 5:30 PM to 5:44 PM, Operational from 5:44 PM to 5:50 PM

  • ResolvedUTC
    This incident has been resolved.
  • MonitoringUTC

    We've identified the root cause to be a transient issue with GitHub's control plane that lasted for ~15 minutes. GitHub’s Actions broker service began rejecting session creation requests from our runners. While our runners were provisioned successfully, they were being rejected by the GitHub Actions broker service, causing the runner to exit without picking up a job.

    This led to job adoption observed delays. The issue resolved on its own as GitHub’s broker service recovered. We also manually requeued jobs that were affected.


    We're seeing recovery in job adoption and queue times are no longer elevated. We will continue to monitor.

  • InvestigatingUTC

    We are seeing some delays in jobs being adopted. We are currently investigating this incident.