Status | Blacksmith - Delays in jobs being assigned to our runners, and dropped webhooks – Incident details

All systems operational

Delays in jobs being assigned to our runners, and dropped webhooks

Resolved
Partial outage
Started 29 days agoLasted about 3 hours

Affected

Blacksmith Managed Runners

Operational from 1:51 PM to 4:42 PM

Github → Actions

Github → API Requests

Updates
  • Resolved
    Resolved
    This incident has been resolved.
  • Monitoring
    Monitoring

    Update from GitHub:

    "Webhooks is experiencing degraded performance. We are continuing to investigate."

    "We have identified a configuration issue that we believe is the source of the Action workflow job delays and page latency increases. We are continuing to investigate and mitigate the issue."

  • Identified
    Identified

    GitHub has declared an incident of their own: https://www.githubstatus.com/incidents/dk61qxd21mtl

  • Investigating
    Investigating

    We're noticing a significantly higher-than-usual delay in jobs being assigned to our runners, and are seeing a small percentage of webhook notifications get completely dropped.