blacksmith - Increased queue times in jobs being assigned to our runner – Incident details

Increased queue times in jobs being assigned to our runner

ResolvedDegraded performance1 hour 8 minutes

Affected

Blacksmith Managed Runners

API

Operational from 3:52 AM to 4:37 AM, Degraded performance from 4:37 AM to 5:01 AM

Website

Github → Actions

Operational from 3:52 AM to 5:01 AM

  • ResolvedUTC

    All services are working as expected.

  • MonitoringUTC

    Blacksmith runners are adopting jobs normally now. A network misconfiguration resulted in some of our services seeing degraded performance. We are continuing to monitor the recovery of our website and dashboard.

  • IdentifiedUTC

    We have identified the issue that is preventing our runners from adopting jobs. We have deployed a fix and expect complete recovery shortly.

  • InvestigatingUTC

    GitHub is not assigning jobs to some of our runners, we are monitoring the situation.