blacksmith - Jobs not being picked up – Incident details

Jobs not being picked up

ResolvedDegraded performance5 hours 24 minutes

Affected

Blacksmith Managed Runners

  • ResolvedUTC

    This incident has been resolved.

  • UpdateUTC

    Jobs are being adopted normally. Queue times remain elevated

  • MonitoringUTC

    We implemented a fix and are currently monitoring the result. Jobs are being picked up. Queue times remain elevated.

  • IdentifiedUTC

    We're experiencing issues with job adoption and are working to identify the cause.

  • UpdateUTC

    We implemented a fix and are currently monitoring the result. Jobs are being picked up. Queue times remain elevated.

  • MonitoringUTC

    We're receiving errors from an external service that is causing a degradation in our feature flagging service. We're rolling out a mitigation and will monitor the results.

  • IdentifiedUTC

    We are continuing to work on a fix.

  • InvestigatingUTC

    We saw a degradation in our control plane and are working on a fix. Jobs will experience longer pickup times.