blacksmith - Extended Queueing in Action Runners – Incident details

Extended Queueing in Action Runners

ResolvedDegraded performance20 minutes

Affected

Blacksmith Managed Runners

Degraded performance from 5:45 PM to 6:05 PM

US West

Degraded performance from 5:45 PM to 6:05 PM

us-west ARM

Degraded performance from 5:45 PM to 6:05 PM

us-west x86

Degraded performance from 5:45 PM to 6:05 PM

EU West

Degraded performance from 5:45 PM to 6:05 PM

eu-west x86

Degraded performance from 5:45 PM to 6:05 PM

  • ResolvedUTC

    We have implemented some mitigations and queue times are back to normal. The incident has been resolved.

  • InvestigatingUTC

    We are currently experiencing elevated tail latencies for certain customers due to job prioritization. We are noticing jobs are taking upwards of 10m to adopt in a few cases. We are looking at options to alleviate queueing.