blacksmith - Jobs not getting picked up – Incident details

All systems operational

Jobs not getting picked up

Resolved
Major outage
Started 2 days agoLasted about 2 hours

Affected

Blacksmith Managed Runners

Partial outage from 1:18 AM to 1:51 AM, Operational from 1:51 AM to 2:11 AM, Degraded performance from 2:11 AM to 2:45 AM, Operational from 2:45 AM to 2:53 AM

us-west ARM

Partial outage from 1:18 AM to 1:51 AM, Operational from 1:51 AM to 2:11 AM, Degraded performance from 2:11 AM to 2:45 AM, Operational from 2:45 AM to 2:53 AM

us-west x86

Partial outage from 1:18 AM to 1:51 AM, Operational from 1:51 AM to 2:11 AM, Degraded performance from 2:11 AM to 2:45 AM, Operational from 2:45 AM to 2:53 AM

eu-central ARM

Partial outage from 1:18 AM to 1:51 AM, Operational from 1:51 AM to 2:11 AM, Degraded performance from 2:11 AM to 2:45 AM, Operational from 2:45 AM to 2:53 AM

eu-central x86

Partial outage from 1:18 AM to 1:51 AM, Operational from 1:51 AM to 2:11 AM, Degraded performance from 2:11 AM to 2:45 AM, Operational from 2:45 AM to 2:53 AM

eu-west x86

Partial outage from 1:18 AM to 1:51 AM, Operational from 1:51 AM to 2:11 AM, Degraded performance from 2:11 AM to 2:45 AM, Operational from 2:45 AM to 2:53 AM

Updates
  • Resolved
    UTC
    Resolved
    This incident has been resolved.
  • Update
    UTC
    Update

    Job adoption is now fully operational.

  • Update
    UTC
    Update

    There is still a delay in job adoption times as we continue to recover.

  • Monitoring
    UTC
    Monitoring

    Jobs adoption has recovered and are operational.

  • Identified
    UTC
    Identified

    We've mitigated the root cause and jobs are resuming to run. Some jobs may still be delayed to start as we catch up with the job backlog.

  • Investigating
    UTC
    Investigating

    We're currently experiencing an outage of our control plane. GitHub job adoption and execution are affected, as well as dashboard access.