blacksmith - US west cache failure – Incident details

This incident is resolved. GitHub Actions cache operations for jobs in us-west have returned to normal, and any jobs that failed during the incident can be re-run.

Cache errors for jobs in us-west have largely subsided. As we complete recovery, some jobs may see a one-time cache miss on their next run. We are monitoring and will provide an update within the next hour.

We have restored the cache infrastructure in us-west, and cache failures have decreased but are not yet back to normal. Jobs in us-west may still see some cache restores and saves fail and fall back to a full install, while the earlier delayed job starts have cleared and other regions are unaffected. We are bringing additional cache capacity online to fully restore service.

We have identified the cause of the failures, and we are implementing a fix to restore cache service. Customers with jobs in us-west may still see cache restores and saves fail and fall back to a full install, which can make those jobs run longer or fail, while jobs in other regions are not affected.