blacksmith - Hanging apt package installs on us-west runners due to an upstream mirror issue – Incident details

Hanging apt package installs on us-west runners due to an upstream mirror issue

ResolvedDegraded performance8 hours 20 minutes

Affected

Blacksmith Managed Runners

Degraded performance from 3:30 AM to 10:33 AM, Partial outage from 10:33 AM to 11:26 AM, Degraded performance from 11:26 AM to 11:50 AM

US West

Degraded performance from 3:30 AM to 10:33 AM, Partial outage from 10:33 AM to 11:26 AM, Degraded performance from 11:26 AM to 11:50 AM

us-west x86

Degraded performance from 3:30 AM to 10:33 AM, Partial outage from 10:33 AM to 11:26 AM, Degraded performance from 11:26 AM to 11:50 AM

  • ResolvedUTC
    This incident has been resolved.
  • MonitoringUTC

    We implemented a fix and are seeing improvements. We are continuing to monitor the result.

  • UpdateUTC

    We are currently deploying a mitigation

  • IdentifiedUTC

    We have identified the affected mirror and are implementing a fix

  • InvestigatingUTC
    We are currently investigating this incident.