blacksmith - Service degradation in upstream GHCR registries – Incident details

All systems operational

Service degradation in upstream GHCR registries

Resolved
Degraded performance
Started 18 days agoLasted about 2 hours

Affected

Github → API Requests

Degraded performance from 2:09 PM to 3:43 PM, Operational from 3:43 PM to 4:30 PM

Updates
  • Resolved
    UTC
    Resolved

    This incident has been resolved. GitHub connectivity from our EU regions was degraded from ~12:45 to 16:15 UTC, affecting checkouts, ghcr.io, and some eu-west job starts. Normal since 16:15. Affected jobs can be re-run.

  • Monitoring
    UTC
    Monitoring

    Connections to github.com and ghcr.io from our eu-west, eu-central, and us-east regions have returned to normal as of approximately 15:00 UTC, so checkouts are completing at normal speed and login failures to GitHub Container Registry have dropped to baseline levels. We are continuing to monitor, and re-running any jobs that failed during this period should succeed.

  • Update
    UTC
    Update

    We are seeing evidence that Git Checkouts are also affected by this. We are seeing high TCP retransmit rates into the EU GitHub loadbalancer. Checkouts in the EU West, EU Central, US East regions are affected.

  • Investigating
    UTC
    Investigating

    We are currently observing higher rates of errors for the upstream GHCR registries which may indicate an undeclared GitHub incident. We are currently monitoring.