Status | Blacksmith - Blacksmith control plane is unresponsive. – Incident details

Blacksmith Managed Runners experiencing degraded performance

Blacksmith control plane is unresponsive.

Resolved
Major outage
Started 17 days agoLasted about 3 hours

Affected

Blacksmith Managed Runners

Major outage from 5:07 PM to 5:33 PM, Partial outage from 5:33 PM to 6:46 PM, Operational from 6:46 PM to 8:05 PM

Incremental Docker Builders

Major outage from 5:07 PM to 5:33 PM, Partial outage from 5:33 PM to 6:46 PM, Operational from 6:46 PM to 8:05 PM

API

Major outage from 5:07 PM to 5:33 PM, Partial outage from 5:33 PM to 6:46 PM, Operational from 6:46 PM to 8:05 PM

Website

Operational from 5:07 PM to 6:38 PM, Major outage from 6:38 PM to 6:46 PM, Operational from 6:46 PM to 8:05 PM

Updates
  • Resolved
    Resolved

    This incident has been resolved, queue times have returned to normal.

  • Monitoring
    Monitoring

    We are seeing some recovery and the system is working through the backlog of the queue. You may continue to see high queue times until the backlog is cleared. We are actively monitoring the situation.

  • Update
    Update

    We're continuing to investigate the slowness to some of our control plane endpoints. Users may continue to notice longer than normal queue times. We are actively investigating what is causing this degradation to alleviate the effect on customer CI jobs.

  • Identified
    Identified
    Interactions with our control plane remain degraded. We are seeing customer jobs getting picked up but users might see longer than normal queue times and longer load times on our dashboard.
  • Investigating
    Investigating
    We are currently investigating this incident.