blacksmith - GitHub Actions jobs losing connection to the GitHub control plane – Incident details

All systems operational

GitHub Actions jobs losing connection to the GitHub control plane

Resolved
Degraded performance
Started over 1 year agoLasted about 1 hour

Affected

Blacksmith Managed Runners

API

Operational from 8:27 PM to 9:42 PM

Github → Actions

Operational from 8:27 PM to 9:42 PM

Updates
  • Resolved
    UTC
    Resolved

    The bug was in the part of our system that allocates static IPs for our runners, but had a wider blast radius that was causing other VMs to get terminated pre-emptively -- the fix has been rolled to the fleet and we're monitoring the incident.

  • Investigating
    UTC
    Investigating

    We're seeing some number of GitHub actions jobs lose their connection to the GitHub control plane. We are currently investigating this incident.