Status | Blacksmith - Notice history

This incident has been resolved, and job pickup has returned to normal.

We have requeued the webhooks that were not processed during the incident, and any affected jobs should start shortly.

We have deployed a fix and are monitoring recovery, and will provide another update within the next 30 minutes.

We have implemented some mitigations and queue times are back to normal. The incident has been resolved.

We are currently scanning for any missed jobs during this degradation period and ensuring that they are being dispatched.

We are seeing job adoption return to normal latencies across all regions.

We are still watching over the job queue draining in one of our regions (eu-central). Customers in this region will see a delay for their jobs to be picked up.

Job adoption in us-west and eu-west have returned to normal. Customers in eu-central may still see delayed job starts while we clear the remaining queue backlog. We have identified the cause of the Bazel caching issue and are now implementing a fix. We will provide another update within the next hour.

Cache and sticky disk operations are healthy again. Bazel caching is not yet operational but being actively investigated by our team. The remaining impact is a backlog of queued jobs that we are actively draining, so some customers may still see delayed job starts until the queue drains.

Our primary Redis instance, which backs our control plane, hit a saturation point, leading to a feedback loop of load. The initial cause was a burst of deliveries of delayed webhooks from GitHub, and our reconciliation systems added further load to the control plane, making things worse.

We have improved load balancing by spreading this workload across multiple Redis instances, and our control plane has fully recovered. The remaining impact is a backlog of queued jobs that we are actively draining, so some customers may still see delayed job starts until the queue clears. We will continue monitoring and update with our findings in the next 30 minutes.

We have applied a change to our backend systems and metrics are showing partial improvement, though not yet back to baseline. Customers can still expect degraded cache interactions while we continue to investigate the root cause. We will provide an update within the next 30 minutes.

We have rolled out fixes to our workload to bring database query latency back to baseline. The degradation has now moved to other parts of our stack. We are continuing to investigate the root cause here. Customers can still expect to see degraded cache interactions. We will provide an update within the next 30 minutes.

We are continuing to investigate degradation in our control plane services. This is causing a large % of caching and stickydisk requests to fail, which in turn cause customer jobs to run much slower than their baseline. The slower runs are exacerbating queueing of new jobs that are coming in to the system. We are actively investigating and rolling out fixes to unblock customer jobs.

We are continuing to investigate the delays in webhook processing on our control plane. Customer's may also see some degradation with caching and sticky disk components.

Queueing improving and currently monitoring.

We are have mitigated the issue on our end, but customers may experience some queueing until we catch up.

Our team have mitigated this and are now seeing job pick up times return to normal as we monitor

We implemented a fix and jobs are being picked up as normal. This incident is now closed.

Our engineers are currently implementing a mitigation. Thank you for your patience.

We have identified the issue as GitHub sending us a high cardinality of malformed webhooks missing critical pieces of information in the payloads. We are working on a patch to work around this as we wait for GitHub to fix the upstream issue.

We believe this is related to an issue with Github Webhooks and are still investigating.

We're continuing to monitor the situation. There's still a slight delay in jobs, but cancelling and rerunning the jobs should resolve the issue.

Continuing to monitor improvements from deployment.

We've deployed mitigations and are observing improvements.

The cluster expansion is complete. Our testing shows that cache interactions are no longer degraded. We're monitoring the cluster to confirm complete recovery.

We're currently adding more capacity to the cluster. There will be a few minutes of slow cache interactions while the operation completes.

We have identified a bad host in the fleet and are working on replacing it. Client cache interactions are recovering as we rollout this fix.