Status | Blacksmith - GitHub Actions jobs losing connection to the GitHub control plane – Incident details

Blacksmith Managed Runners experiencing degraded performance

GitHub Actions jobs losing connection to the GitHub control plane

Resolved
Degraded performance
Started 5 months agoLasted about 1 hour

Affected

Blacksmith Managed Runners

Degraded performance from 8:27 PM to 9:42 PM

API

Operational from 8:27 PM to 9:42 PM

Github → Actions

Updates
  • Resolved
    Resolved

    The bug was in the part of our system that allocates static IPs for our runners, but had a wider blast radius that was causing other VMs to get terminated pre-emptively -- the fix has been rolled to the fleet and we're monitoring the incident.

  • Investigating
    Investigating

    We're seeing some number of GitHub actions jobs lose their connection to the GitHub control plane. We are currently investigating this incident.