blacksmith - Job adoption delays – Incident details

All systems operational

Job adoption delays

Resolved
Major outage
Started 4 months agoLasted 1 hour 3 minutes

Affected

Blacksmith Managed Runners

Operational from 12:45 PM to 1:01 PM, Partial outage from 1:01 PM to 1:14 PM, Operational from 1:01 PM to 1:14 PM

US West

Operational from 12:45 PM to 1:01 PM, Partial outage from 1:01 PM to 1:14 PM, Operational from 1:01 PM to 1:27 PM

us-west ARM

Operational from 12:45 PM to 1:01 PM, Partial outage from 1:01 PM to 1:14 PM, Operational from 1:01 PM to 1:27 PM

us-west x86

Operational from 12:45 PM to 1:01 PM, Partial outage from 1:01 PM to 1:14 PM, Operational from 1:01 PM to 1:27 PM

EU Central

Operational from 12:45 PM to 1:14 PM

eu-central ARM

Operational from 12:45 PM to 1:14 PM

Updates
  • Update
    UTC
    Update
    This incident has been resolved.
  • Resolved
    UTC
    Resolved

    We implemented a fix and jobs are being picked up as normal. This incident is now closed.

  • Monitoring
    UTC
    Monitoring

    Our engineers are currently implementing a mitigation. Thank you for your patience.

  • Identified
    UTC
    Identified

    We have identified the issue as GitHub sending us a high cardinality of malformed webhooks missing critical pieces of information in the payloads. We are working on a patch to work around this as we wait for GitHub to fix the upstream issue.

  • Update
    UTC
    Update

    We believe this is related to an issue with Github Webhooks and are still investigating.

  • Investigating
    UTC
    Investigating

    We are receiving reports of job adoption delays, we are currently investigating.