Status | Blacksmith - Delays in Job Adoption – Incident details

Delays in Job Adoption

Resolved
Operational
Started 4 days agoLasted 19 minutes

Affected

Blacksmith Managed Runners

Degraded performance from 5:30 PM to 5:44 PM, Operational from 5:44 PM to 5:50 PM

US ARM

Degraded performance from 5:30 PM to 5:44 PM, Operational from 5:44 PM to 5:50 PM

US X86

Degraded performance from 5:30 PM to 5:44 PM, Operational from 5:44 PM to 5:50 PM

Updates
  • Resolved
    Resolved
    This incident has been resolved.
  • Monitoring
    Monitoring

    We've identified the root cause to be a transient issue with GitHub's control plane that lasted for ~15 minutes. GitHub’s Actions broker service began rejecting session creation requests from our runners. While our runners were provisioned successfully, they were being rejected by the GitHub Actions broker service, causing the runner to exit without picking up a job.

    This led to job adoption observed delays. The issue resolved on its own as GitHub’s broker service recovered. We also manually requeued jobs that were affected.


    We're seeing recovery in job adoption and queue times are no longer elevated. We will continue to monitor.

  • Investigating
    Investigating

    We are seeing some delays in jobs being adopted. We are currently investigating this incident.