blacksmith - Hanging apt package installs on us-west runners due to an upstream mirror issue – Incident details

All systems operational

Hanging apt package installs on us-west runners due to an upstream mirror issue

Resolved
Degraded performance
Started 17 days agoLasted about 8 hours

Affected

Blacksmith Managed Runners

Degraded performance from 3:30 AM to 10:33 AM, Partial outage from 10:33 AM to 11:26 AM, Degraded performance from 11:26 AM to 11:50 AM

US West

Degraded performance from 3:30 AM to 10:33 AM, Partial outage from 10:33 AM to 11:26 AM, Degraded performance from 11:26 AM to 11:50 AM

us-west x86

Degraded performance from 3:30 AM to 10:33 AM, Partial outage from 10:33 AM to 11:26 AM, Degraded performance from 11:26 AM to 11:50 AM

Updates
  • Resolved
    UTC
    Resolved
    This incident has been resolved.
  • Monitoring
    UTC
    Monitoring

    We implemented a fix and are seeing improvements. We are continuing to monitor the result.

  • Update
    UTC
    Update

    We are currently deploying a mitigation

  • Identified
    UTC
    Identified

    We have identified the affected mirror and are implementing a fix

  • Investigating
    UTC
    Investigating
    We are currently investigating this incident.