Status | Blacksmith - Degradation in US West storage cluster – Incident details

All systems operational

Degradation in US West storage cluster

Resolved
Degraded performance
Started 3 days agoLasted about 4 hours

Affected

Incremental Docker Builders

Degraded performance from 7:56 PM to 9:59 PM, Operational from 9:59 PM to 10:28 PM, Degraded performance from 10:28 PM to 11:32 PM

Updates
  • Resolved
    Resolved
    This incident has been resolved.
  • Monitoring
    Monitoring

    Fix has been deployed, and we are currently monitoring the state of the cluster.

  • Identified
    Identified

    We're about to deploy a fix, this will result in a few minutes of downtime and all stickydisk artifacts being cleared.

  • Monitoring
    Monitoring

    We are still seeing signs of degradation in bursts as the storage cluster trends towards recovery. We will continue monitoring the situation and provide updates here.

  • Resolved
    Resolved

    This incident has been resolved. We had a sudden surge of usage that required us to scale up our storage cluster.

  • Investigating
    Investigating
    We are currently investigating this incident.