Status | Blacksmith - Webhook Service Outage – Incident details

All systems operational

Webhook Service Outage

Resolved
Major outage
Started 3 days agoLasted about 3 hours

Affected

Blacksmith Managed Runners

Major outage from 8:39 PM to 8:42 PM, Degraded performance from 8:42 PM to 11:46 PM

EU ARM

Major outage from 8:39 PM to 8:42 PM, Degraded performance from 8:42 PM to 11:46 PM

EU X86

Major outage from 8:39 PM to 8:42 PM, Degraded performance from 8:42 PM to 11:46 PM

US ARM

Major outage from 8:39 PM to 8:42 PM, Degraded performance from 8:42 PM to 11:46 PM

US X86

Major outage from 8:39 PM to 8:42 PM, Degraded performance from 8:42 PM to 11:46 PM

EU-WEST x86

Major outage from 8:39 PM to 8:42 PM, Degraded performance from 8:42 PM to 11:46 PM

Updates
  • Resolved
    Resolved

    We have been running into load issues in our database layer that's causing sporadic instability in job adoption times. We have been actively working with our database provider to improve our systems to handle the recently increased load. We are planning proactive maintenance later this week to address this further.

  • Monitoring
    Monitoring
    We implemented a fix and are currently monitoring the result.
  • Investigating
    Investigating
    We are currently investigating this incident.