blacksmith - Backend Degradation causing job slowness and caching failures – Incident details

All systems operational

Show current status

Backend Degradation causing job slowness and caching failures

ResolvedDegraded performance1 hour 54 minutes

Affected

  • ResolvedUTC

    This incident has been resolved.

  • UpdateUTC

    This incident is resolved. Job start times and cache operations returned to normal in all regions, and we are continuing to monitor. Jobs that failed during the incident can be re-run.

  • UpdateUTC

    Caching has recovered, and job queue times across EU regions have recovered. Queue times for larger jobs (16 and 32 vcpu jobs) in US west and US east continue to remain elevated. We are continuing to monitor for any regressions.

  • MonitoringUTC

    Our mitigation has taken effect and job start times and cache operations are improving. Customers may still see some delayed job starts and cache failures while recovery completes. We will provide an update within the next 30 minutes.

  • IdentifiedUTC

    We have applied a fix and are monitoring its effect. Customers may still see delayed job starts and failing cache operations while recovery completes. We will provide an update within the next 30 minutes.

  • InvestigatingUTC

    Jobs across all regions are taking longer to start and cache operations are failing. We are continuing to investigate the root cause.