Delays with job adoption
Affected
- Postmortem · UTC
What happened: A bug in our job reconciliation system (which ensures we spin up VMs to run your CI jobs even if GitHub fails to send a webhook) caused us to overprovision VMs, spinning up 110% more VMs than needed. This additional load led to increased queue times and made it harder for the system to recover quickly due to delays in draining the additional VMs.
What we’ve done since:
Fixed the underlying bug
Improved how we drain the queue under load
Added safeguards to better handle similar failure modes in the future
- Resolved · UTC
This incident has been resolved, queue times are returning to normal.
- Identified · UTC
We are running into fleet capacity issues in our amd64 cluster, and are looking into mitigations.
- Investigating · UTC
We're seeing delays in job adoption times for AMD64 jobs across both our EU and U.S. regions. We're currently investigating this issue.
