blacksmith - Increased queue times in jobs being assigned to our runner – Incident details

All systems operational

Increased queue times in jobs being assigned to our runner

Resolved
Degraded performance
Started over 1 year agoLasted about 1 hour

Affected

Blacksmith Managed Runners

API

Operational from 3:52 AM to 4:37 AM, Degraded performance from 4:37 AM to 5:01 AM

Website

Github → Actions

Operational from 3:52 AM to 5:01 AM

Updates
  • Resolved
    UTC
    Resolved

    All services are working as expected.

  • Monitoring
    UTC
    Monitoring

    Blacksmith runners are adopting jobs normally now. A network misconfiguration resulted in some of our services seeing degraded performance. We are continuing to monitor the recovery of our website and dashboard.

  • Identified
    UTC
    Identified

    We have identified the issue that is preventing our runners from adopting jobs. We have deployed a fix and expect complete recovery shortly.

  • Investigating
    UTC
    Investigating

    GitHub is not assigning jobs to some of our runners, we are monitoring the situation.