blacksmith - GitHub outage affecting job failures and dashboard errors – Incident details

All systems operational

GitHub outage affecting job failures and dashboard errors

Resolved
Major outage
Started 13 days agoLasted about 7 hours

Affected

Github → API Requests

Operational from 1:43 PM to 8:40 PM

Github → Webhooks

Operational from 1:43 PM to 8:40 PM

Dashboard

Major outage from 1:43 PM to 4:56 PM, Operational from 4:56 PM to 8:40 PM

Updates
  • Resolved
    UTC
    Resolved
    This incident has been resolved.
  • Update
    UTC
    Update

    We are no longer seeing upstream errors, and are continuing to monitor impact of the upstream outage.

  • Update
    UTC
    Update

    We are still seeing intermitting GitHub API errors at a low rate.

  • Monitoring
    UTC
    Monitoring

    We're seeing signs of GitHub recovery. The dashboard is now loading and jobs should be running again. We are monitoring the recovery.

  • Update
    UTC
    Update

    Upstream incident has been declared: https://www.githubstatus.com/incidents/zkxwbgr0cnmx

    We are also seeing elevated error rates in jobs as they hit upstream GitHub errors. Job adoption times are also affected and are delayed.

    We are monitoring and are looking at potential mitigations.

  • Identified
    UTC
    Identified

    The Blacksmith Dashboard is unable to load. We've identified the root cause to upstream 503s being returned from GitHub on permission-check requests.