Status | Blacksmith - Webhook Service Outage – Incident details

All systems operational

Webhook Service Outage

Resolved
Major outage
Started 2 months agoLasted about 3 hours

Affected

Blacksmith Managed Runners

Major outage from 8:39 PM to 8:42 PM, Degraded performance from 8:42 PM to 11:46 PM

us-west ARM

Major outage from 8:39 PM to 8:42 PM, Degraded performance from 8:42 PM to 11:46 PM

us-west x86

Major outage from 8:39 PM to 8:42 PM, Degraded performance from 8:42 PM to 11:46 PM

eu-central ARM

Major outage from 8:39 PM to 8:42 PM, Degraded performance from 8:42 PM to 11:46 PM

eu-central x86

Major outage from 8:39 PM to 8:42 PM, Degraded performance from 8:42 PM to 11:46 PM

eu-west x86

Major outage from 8:39 PM to 8:42 PM, Degraded performance from 8:42 PM to 11:46 PM

Updates
  • Resolved
    UTC
    Resolved

    We have been running into load issues in our database layer that's causing sporadic instability in job adoption times. We have been actively working with our database provider to improve our systems to handle the recently increased load. We are planning proactive maintenance later this week to address this further.

  • Monitoring
    UTC
    Monitoring
    We implemented a fix and are currently monitoring the result.
  • Investigating
    UTC
    Investigating
    We are currently investigating this incident.