Status | Blacksmith - Notice history

We implemented a fix and jobs are being picked up as normal. This incident is now closed.

Our engineers are currently implementing a mitigation. Thank you for your patience.

We have identified the issue as GitHub sending us a high cardinality of malformed webhooks missing critical pieces of information in the payloads. We are working on a patch to work around this as we wait for GitHub to fix the upstream issue.

We believe this is related to an issue with Github Webhooks and are still investigating.

Speeds have returned to normal and this incident has been resolved.

us-west still experiencing some mild queueing, and eu-central and eu-west has recovered.

We are continuing to work on a fix for this incident. eu-central and us-west still experiencing some queueing, and eu-west has recovered.

We're continuing to monitor the situation. There's still a slight delay in jobs, but cancelling and rerunning the jobs should resolve the issue.

Continuing to monitor improvements from deployment.

We've deployed mitigations and are observing improvements.

The cluster expansion is complete. Our testing shows that cache interactions are no longer degraded. We're monitoring the cluster to confirm complete recovery.

We're currently adding more capacity to the cluster. There will be a few minutes of slow cache interactions while the operation completes.

We have identified a bad host in the fleet and are working on replacing it. Client cache interactions are recovering as we rollout this fix.

We are continuing to see recovery in us-west

This incident has been resolved.

The re-queuing is still in progress however new jobs are being picked up without delay.

We have completed the DB operation and currently re-queuing jobs. This may take ~15 minutes

We needed to perform an emergency maintenance operation in our storage cluster. We are observing the recovery and seeing docker build speeds improve.