blacksmith - Job adoption delays – Incident details

Job adoption delays

ResolvedMajor outage1 hour 3 minutes

Affected

Github → Actions

Operational from 12:45 PM to 1:48 PM

Github → Webhooks

Operational from 12:45 PM to 1:48 PM

  • UpdateUTC
    This incident has been resolved.
  • ResolvedUTC

    We implemented a fix and jobs are being picked up as normal. This incident is now closed.

  • MonitoringUTC

    Our engineers are currently implementing a mitigation. Thank you for your patience.

  • IdentifiedUTC

    We have identified the issue as GitHub sending us a high cardinality of malformed webhooks missing critical pieces of information in the payloads. We are working on a patch to work around this as we wait for GitHub to fix the upstream issue.

  • UpdateUTC

    We believe this is related to an issue with Github Webhooks and are still investigating.

  • InvestigatingUTC

    We are receiving reports of job adoption delays, we are currently investigating.