blacksmith - Degradation in US West storage cluster – Incident details

Degradation in US West storage cluster

ResolvedDegraded performance3 hours 35 minutes

Affected

Incremental Docker Builders

  • ResolvedUTC
    This incident has been resolved.
  • MonitoringUTC

    Fix has been deployed, and we are currently monitoring the state of the cluster.

  • IdentifiedUTC

    We're about to deploy a fix, this will result in a few minutes of downtime and all stickydisk artifacts being cleared.

  • MonitoringUTC

    We are still seeing signs of degradation in bursts as the storage cluster trends towards recovery. We will continue monitoring the situation and provide updates here.

  • ResolvedUTC

    This incident has been resolved. We had a sudden surge of usage that required us to scale up our storage cluster.

  • InvestigatingUTC
    We are currently investigating this incident.