blacksmith - Network degradation for runners in our US region – Incident details

All systems operational

Network degradation for runners in our US region

Resolved
Degraded performance
Started over 1 year agoLasted about 13 hours

Affected

Blacksmith Managed Runners

Github → API Requests

Operational from 1:17 AM to 2:41 PM

Updates
  • Resolved
    UTC
    Resolved

    This incident has been resolved, network health is back to normal in our US fleet.

  • Monitoring
    UTC
    Monitoring

    We have been informed that the upstream networking issue has now been resolved and service should be returning to normal. We are working with our vendor to better understand how to avoid this in the future.

  • Update
    UTC
    Update

    Our vendor is actively working on a fix for this network degradation. In the mean time we are removing all the affected servers from our fleet to help bring service back to normal.

  • Identified
    UTC
    Identified

    We are specifically seeing packets being dropped when downloading from api.github.com. We continue to look into the source of the degradation.

  • Investigating
    UTC
    Investigating

    We are currently investigating reports from an upstream vendor of a degraded network stack in our US region. This could manifest as hanging GitHub Actions runners or slow/stuck steps in the workflow.