blacksmith - Blacksmith control plane is unresponsive. – Incident details

All systems operational

Blacksmith control plane is unresponsive.

Resolved
Major outage
Started 11 months agoLasted about 3 hours

Affected

Blacksmith Managed Runners

Incremental Docker Builders

API

Major outage from 5:07 PM to 5:33 PM, Partial outage from 5:33 PM to 6:46 PM, Operational from 6:46 PM to 8:05 PM

Website

Updates
  • Resolved
    UTC
    Resolved

    This incident has been resolved, queue times have returned to normal.

  • Monitoring
    UTC
    Monitoring

    We are seeing some recovery and the system is working through the backlog of the queue. You may continue to see high queue times until the backlog is cleared. We are actively monitoring the situation.

  • Update
    UTC
    Update

    We're continuing to investigate the slowness to some of our control plane endpoints. Users may continue to notice longer than normal queue times. We are actively investigating what is causing this degradation to alleviate the effect on customer CI jobs.

  • Identified
    UTC
    Identified
    Interactions with our control plane remain degraded. We are seeing customer jobs getting picked up but users might see longer than normal queue times and longer load times on our dashboard.
  • Investigating
    UTC
    Investigating
    We are currently investigating this incident.