blacksmith - Delays with job adoption – Incident details

Delays with job adoption

ResolvedDegraded performance5 hours 37 minutes

Affected

Blacksmith Managed Runners

  • PostmortemUTC

    What happened: A bug in our job reconciliation system (which ensures we spin up VMs to run your CI jobs even if GitHub fails to send a webhook) caused us to overprovision VMs, spinning up 110% more VMs than needed. This additional load led to increased queue times and made it harder for the system to recover quickly due to delays in draining the additional VMs.

    What we’ve done since:

    • Fixed the underlying bug

    • Improved how we drain the queue under load

    • Added safeguards to better handle similar failure modes in the future

  • ResolvedUTC

    This incident has been resolved, queue times are returning to normal.

  • IdentifiedUTC

    We are running into fleet capacity issues in our amd64 cluster, and are looking into mitigations.

  • InvestigatingUTC

    We're seeing delays in job adoption times for AMD64 jobs across both our EU and U.S. regions. We're currently investigating this issue.