Jobs not being picked up

Resolved

Degraded performance

Started 10 months agoOctober 28, 2025Lasted about 5 hoursOctober 28, 20257:57 PMOctober 29, 20251:22 AMUTC

Affected

Blacksmith Managed Runners

Updates

Resolved
October 29, 2025 at 1:22 AMUTC
Resolved
October 29, 2025 at 1:22 AMUTC
This incident has been resolved.
Update
October 29, 2025 at 12:11 AMUTC
Update
October 29, 2025 at 12:11 AMUTC
Jobs are being adopted normally. Queue times remain elevated
Monitoring
October 28, 2025 at 10:57 PMUTC
Monitoring
October 28, 2025 at 10:57 PMUTC
We implemented a fix and are currently monitoring the result. Jobs are being picked up. Queue times remain elevated.
Identified
October 28, 2025 at 9:51 PMUTC
Identified
October 28, 2025 at 9:51 PMUTC
We're experiencing issues with job adoption and are working to identify the cause.
Update
October 28, 2025 at 9:32 PMUTC
Update
October 28, 2025 at 9:32 PMUTC
We implemented a fix and are currently monitoring the result. Jobs are being picked up. Queue times remain elevated.
Monitoring
October 28, 2025 at 8:59 PMUTC
Monitoring
October 28, 2025 at 8:59 PMUTC
We're receiving errors from an external service that is causing a degradation in our feature flagging service. We're rolling out a mitigation and will monitor the results.
Identified
October 28, 2025 at 8:28 PMUTC
Identified
October 28, 2025 at 8:28 PMUTC
We are continuing to work on a fix.
Investigating
October 28, 2025 at 7:57 PMUTC
Investigating
October 28, 2025 at 7:57 PMUTC
We saw a degradation in our control plane and are working on a fix. Jobs will experience longer pickup times.

blacksmith - Jobs not being picked up – Incident details