blacksmith - Jobs not starting and dashboard unavailable – Incident details

Jobs not starting and dashboard unavailable

ResolvedMajor outage1 hour 2 minutes

Affected

Dashboard

Major outage from 8:55 PM to 9:02 PM, Operational from 9:02 PM to 9:57 PM

  • ResolvedUTC

    Job pickup has been back to normal in all regions since 21:48 UTC, after we reduced load on the database behind our control plane and rebalanced capacity across regions. Workflow runs that failed or timed out between 20:47 and 21:48 UTC will not retry on their own, so please re-run them.

  • MonitoringUTC

    The Blacksmith dashboard is fully recovered and jobs are being picked up normally in us-east, eu-central, us-central and us-west-2. Jobs in us-west and eu-west are waiting longer than normal to be picked up while we clear the queue that built up during the disruption, and we are rebalancing capacity across regions.

  • IdentifiedUTC

    The Blacksmith dashboard has fully recovered and jobs have been scheduling again since 20:58 UTC, after we restored capacity in the affected backend systems.

    We are working through the jobs that queued during the disruption, so customers in all regions may still see delayed job starts until that clears.

  • InvestigatingUTC

    Jobs are not being scheduled and the Blacksmith dashboard is unavailable, affecting all regions. We have traced the disruption to saturation in part of our backend systems and are continuing to investigate the underlying cause.