blacksmith - GitHub outage affecting job failures and dashboard errors – Incident details

All systems operational

Show current status

GitHub outage affecting job failures and dashboard errors

ResolvedMajor outage6 hours 57 minutes

Affected

Github → API Requests

Operational from 1:43 PM to 8:40 PM

Github → Webhooks

Operational from 1:43 PM to 8:40 PM

Dashboard

Major outage from 1:43 PM to 4:56 PM, Operational from 4:56 PM to 8:40 PM

  • ResolvedUTC
    This incident has been resolved.
  • UpdateUTC

    We are no longer seeing upstream errors, and are continuing to monitor impact of the upstream outage.

  • UpdateUTC

    We are still seeing intermitting GitHub API errors at a low rate.

  • MonitoringUTC

    We're seeing signs of GitHub recovery. The dashboard is now loading and jobs should be running again. We are monitoring the recovery.

  • UpdateUTC

    Upstream incident has been declared: https://www.githubstatus.com/incidents/zkxwbgr0cnmx

    We are also seeing elevated error rates in jobs as they hit upstream GitHub errors. Job adoption times are also affected and are delayed.

    We are monitoring and are looking at potential mitigations.

  • IdentifiedUTC

    The Blacksmith Dashboard is unable to load. We've identified the root cause to upstream 503s being returned from GitHub on permission-check requests.