Status | Blacksmith - Jobs not being picked up – Incident details

All systems operational

Jobs not being picked up

Resolved
Degraded performance
Started 3 months agoLasted about 5 hours

Affected

Blacksmith Managed Runners

Degraded performance from 7:57 PM to 12:11 AM, Operational from 12:11 AM to 1:22 AM

Updates
  • Resolved
    Resolved

    This incident has been resolved.

  • Update
    Update

    Jobs are being adopted normally. Queue times remain elevated

  • Monitoring
    Monitoring

    We implemented a fix and are currently monitoring the result. Jobs are being picked up. Queue times remain elevated.

  • Identified
    Identified

    We're experiencing issues with job adoption and are working to identify the cause.

  • Update
    Update

    We implemented a fix and are currently monitoring the result. Jobs are being picked up. Queue times remain elevated.

  • Monitoring
    Monitoring

    We're receiving errors from an external service that is causing a degradation in our feature flagging service. We're rolling out a mitigation and will monitor the results.

  • Identified
    Identified

    We are continuing to work on a fix.

  • Investigating
    Investigating

    We saw a degradation in our control plane and are working on a fix. Jobs will experience longer pickup times.