blacksmith - Upstream Ubuntu package mirror outage affecting apt – Incident details

All systems operational

Upstream Ubuntu package mirror outage affecting apt

Resolved
Degraded performance
Started 12 days agoLasted about 10 hours
Updates
  • Resolved
    UTC
    Resolved

    This incident has been resolved - Canonical's Ubuntu apt mirrors have recovered and apt installs are succeeding again. We are also rolling out an image change to reduce the impact of upstream mirror outages in future.

  • Update
    UTC
    Update

    We are deploying a mitigation on our side that routes apt around the affected Canonical mirrors. For the time being pointing apt at azure.archive.ubuntu.com in place of archive.ubuntu.com and security.ubuntu.com will unblock affected jobs.

  • Monitoring
    UTC
    Monitoring

    Since 06:30 UTC Canonical has had three outages affecting archive.ubuntu.com, us.archive.ubuntu.com and security.ubuntu.com. Although they have marked these as resolved, we are still seeing intermittent connection failures to those mirrors, which can cause apt-get steps to fail or hang.

    In the meantime, pointing apt at azure.archive.ubuntu.com instead of archive.ubuntu.com and security.ubuntu.com will unblock affected jobs.