Status | Blacksmith - Jobs not being adopted in all regions – Incident details

All systems operational

Jobs not being adopted in all regions

Resolved
Partial outage
Started about 6 hours agoLasted 34 minutes

Affected

Blacksmith Managed Runners

Partial outage from 7:17 PM to 7:20 PM, Degraded performance from 7:20 PM to 7:51 PM

us-west ARM

Partial outage from 7:17 PM to 7:20 PM, Degraded performance from 7:20 PM to 7:51 PM

us-west x86

Partial outage from 7:17 PM to 7:20 PM, Degraded performance from 7:20 PM to 7:51 PM

eu-central ARM

Partial outage from 7:17 PM to 7:20 PM, Degraded performance from 7:20 PM to 7:51 PM

eu-central x86

Partial outage from 7:17 PM to 7:20 PM, Degraded performance from 7:20 PM to 7:51 PM

eu-west x86

Partial outage from 7:17 PM to 7:20 PM, Degraded performance from 7:20 PM to 7:51 PM

Updates
  • Resolved
    UTC
    Resolved

    This incident has been resolved, and job pickup has returned to normal.

  • Update
    UTC
    Update

    We have requeued the webhooks that were not processed during the incident, and any affected jobs should start shortly.

  • Monitoring
    UTC
    Monitoring

    We have deployed a fix and are monitoring recovery, and will provide another update within the next 30 minutes.

  • Investigating
    UTC
    Investigating

    We are seeing webhooks not being accepted by our control plane resulting in jobs not being adopted. We are currently investigating.