blacksmith - Blacksmith control plane is unresponsive. – Incident details

Blacksmith control plane is unresponsive.

ResolvedMajor outage2 hours 58 minutes

Affected

Blacksmith Managed Runners

Incremental Docker Builders

API

Major outage from 5:07 PM to 5:33 PM, Partial outage from 5:33 PM to 6:46 PM, Operational from 6:46 PM to 8:05 PM

Website

  • ResolvedUTC

    This incident has been resolved, queue times have returned to normal.

  • MonitoringUTC

    We are seeing some recovery and the system is working through the backlog of the queue. You may continue to see high queue times until the backlog is cleared. We are actively monitoring the situation.

  • UpdateUTC

    We're continuing to investigate the slowness to some of our control plane endpoints. Users may continue to notice longer than normal queue times. We are actively investigating what is causing this degradation to alleviate the effect on customer CI jobs.

  • IdentifiedUTC
    Interactions with our control plane remain degraded. We are seeing customer jobs getting picked up but users might see longer than normal queue times and longer load times on our dashboard.
  • InvestigatingUTC
    We are currently investigating this incident.