blacksmith - Notice history

This incident has been resolved.

Jobs are being adopted normally. Queue times remain elevated

We implemented a fix and are currently monitoring the result. Jobs are being picked up. Queue times remain elevated.

We're experiencing issues with job adoption and are working to identify the cause.

We implemented a fix and are currently monitoring the result. Jobs are being picked up. Queue times remain elevated.

We're receiving errors from an external service that is causing a degradation in our feature flagging service. We're rolling out a mitigation and will monitor the results.

We are continuing to work on a fix.

Jobs are being picked up. Queue times have increased slightly as we work through affected jobs. We are continuing to monitor.

Jobs are being picked up and we are continuing to monitor.

This incident has been resolved, our control plane is functioning as normal. We are working with our DB provider to understand how to avoid such outages in the future.

Our fixes appear successful. We're continuing to monitor.

We're monitoring our control plane endpoints response as we work toward a fix.

We've isolated the issues to our database provider and are working with them to restore functionality

We are continuing to work on a fix for this incident.

This incident has been resolved, queue times are back to normal. The longer queue times were a combination of an outage with our upstream DB provider and a need for us to scale up our fleet to absorb higher than normal traffic. We are working with our DB provider to prevent such an outage in the future, and have scaled up our fleet to prevent such saturation in the future.

We sincerely apologize for the prolonged queueing that our customers saw today. We take such a prolonged outage very seriously and will work on hardening our systems based on our discoveries from today. If you still see any queued jobs we would recommend canceling and re-triggering jobs to see normal adoption.

Queue times are stabilizing and we're continuing to monitor.

We are monitoring increased queue times in relation to an incident with our database provider and are working towards a fix

We are continuing to monitor increased queue times.

We are continuing to monitor increasing queue times. Our team is working on mitigations to bring times back down to normal.

Increased queue times continue but we are seeing improvement and are monitoring the situation.

We are continuing to investigate increased queue times in the US and are working to resolve this.

We are continuing to investigate increased queue times in the US

This incident has been resolved, queue times have returned to normal.

We are seeing some recovery and the system is working through the backlog of the queue. You may continue to see high queue times until the backlog is cleared. We are actively monitoring the situation.

We're continuing to investigate the slowness to some of our control plane endpoints. Users may continue to notice longer than normal queue times. We are actively investigating what is causing this degradation to alleviate the effect on customer CI jobs.

This incident has been resolved, cache operations are back to normal in the EU.

This incident has been resolved. We will be providing a postmortem to affected customers by tomorrow.

Our upstream ISP identified some issues between them and their network vendor. They have made changes to mitigate these blips while working on a longer term resolution. We will share more details as we receive them. All network operations in Blacksmith runners have returned to normal. We are actively monitoring the situation.

We are being alerted about intermittent degradation in our network stack in one of our regions. This may be sporadically manifesting as slower docker pulls, interactions with certain language mirrors, git checkouts. We are investigating the source of this issue.

This incident has been resolved, we are no longer seeing signs of this flake with GitHub.

Fix has been deployed, and we are currently monitoring the state of the cluster.

We're about to deploy a fix, this will result in a few minutes of downtime and all stickydisk artifacts being cleared.

We are still seeing signs of degradation in bursts as the storage cluster trends towards recovery. We will continue monitoring the situation and provide updates here.

This incident has been resolved. We had a sudden surge of usage that required us to scale up our storage cluster.

This incident has been resolved Docker builds should now be working as expected.

What happened: A bug in our job reconciliation system (which ensures we spin up VMs to run your CI jobs even if GitHub fails to send a webhook) caused us to overprovision VMs, spinning up 110% more VMs than needed. This additional load led to increased queue times and made it harder for the system to recover quickly due to delays in draining the additional VMs.

What we’ve done since:

• Fixed the underlying bug

• Improved how we drain the queue under load

• Added safeguards to better handle similar failure modes in the future

This incident has been resolved, queue times are returning to normal.

We are running into fleet capacity issues in our amd64 cluster, and are looking into mitigations.