blacksmith - Runners and API are unavailable – Incident details

All systems operational

Runners and API are unavailable

Resolved
Major outage
Started 7 months agoLasted about 1 hour

Affected

Blacksmith Managed Runners

Operational from 9:46 AM to 10:42 AM

US West

Operational from 9:46 AM to 10:42 AM

us-west ARM

Operational from 9:46 AM to 10:42 AM

us-west x86

Operational from 9:46 AM to 10:42 AM

EU Central

Operational from 9:46 AM to 10:42 AM

eu-central ARM

Operational from 9:46 AM to 10:42 AM

Updates
  • Resolved
    UTC
    Resolved
    This incident has been resolved.
  • Update
    UTC
    Update

    Systems are recovering & we are currently clearing the job queue backlog that was affected. If your jobs are still showing as queued, please cancel and re-run them to ensure they get picked up.                   

  • Monitoring
    UTC
    Monitoring

    We're seeing recovery and are monitoring closely.

  • Investigating
    UTC
    Investigating

    We're currently experiencing service disruption due to an outage at one of our infrastructure providers Clickhouse, this may cause some disruption to the dashboard and runners -https://statuspage.incident.io/clickhousecloud/incidents/01KG9R3JMK16VXF5CHBDZ4YNC9