Jobs not starting and dashboard unavailable
Affected
Major outage from 8:55 PM to 9:02 PM, Operational from 9:02 PM to 9:57 PM
- Resolved · UTC
Job pickup has been back to normal in all regions since 21:48 UTC, after we reduced load on the database behind our control plane and rebalanced capacity across regions. Workflow runs that failed or timed out between 20:47 and 21:48 UTC will not retry on their own, so please re-run them.
- Monitoring · UTC
The Blacksmith dashboard is fully recovered and jobs are being picked up normally in us-east, eu-central, us-central and us-west-2. Jobs in us-west and eu-west are waiting longer than normal to be picked up while we clear the queue that built up during the disruption, and we are rebalancing capacity across regions.
- Identified · UTC
The Blacksmith dashboard has fully recovered and jobs have been scheduling again since 20:58 UTC, after we restored capacity in the affected backend systems.
We are working through the jobs that queued during the disruption, so customers in all regions may still see delayed job starts until that clears. - Investigating · UTC
Jobs are not being scheduled and the Blacksmith dashboard is unavailable, affecting all regions. We have traced the disruption to saturation in part of our backend systems and are continuing to investigate the underlying cause.
