Status | Blacksmith - Notice history

This incident has been resolved, queue times have returned to normal.

We are seeing some recovery and the system is working through the backlog of the queue. You may continue to see high queue times until the backlog is cleared. We are actively monitoring the situation.

We're continuing to investigate the slowness to some of our control plane endpoints. Users may continue to notice longer than normal queue times. We are actively investigating what is causing this degradation to alleviate the effect on customer CI jobs.

This incident has been resolved, cache operations are back to normal in the EU.

This incident has been resolved. We will be providing a postmortem to affected customers by tomorrow.

Our upstream ISP identified some issues between them and their network vendor. They have made changes to mitigate these blips while working on a longer term resolution. We will share more details as we receive them. All network operations in Blacksmith runners have returned to normal. We are actively monitoring the situation.

We are being alerted about intermittent degradation in our network stack in one of our regions. This may be sporadically manifesting as slower docker pulls, interactions with certain language mirrors, git checkouts. We are investigating the source of this issue.

This incident has been resolved, we are no longer seeing signs of this flake with GitHub.

Fix has been deployed, and we are currently monitoring the state of the cluster.

We're about to deploy a fix, this will result in a few minutes of downtime and all stickydisk artifacts being cleared.

We are still seeing signs of degradation in bursts as the storage cluster trends towards recovery. We will continue monitoring the situation and provide updates here.

This incident has been resolved. We had a sudden surge of usage that required us to scale up our storage cluster.

This incident has been resolved Docker builds should now be working as expected.

What happened: A bug in our job reconciliation system (which ensures we spin up VMs to run your CI jobs even if GitHub fails to send a webhook) caused us to overprovision VMs, spinning up 110% more VMs than needed. This additional load led to increased queue times and made it harder for the system to recover quickly due to delays in draining the additional VMs.

What we’ve done since:

• Fixed the underlying bug

• Improved how we drain the queue under load

• Added safeguards to better handle similar failure modes in the future

This incident has been resolved, queue times are returning to normal.

We are running into fleet capacity issues in our amd64 cluster, and are looking into mitigations.

This incident has been resolved by GitHub.

This incident has been resolved, queue times are back to normal.

This incident has been resolved, we've deployed a mitigation that has stabilized our control plane.

Users may also see degraded performance interacting with our cache endpoints, we are investigating the incident.

This incident has been resolved we will be gradually re-enabling transparent caching for all orgs.

We are implementing a fix to re-enable transparent caching for all orgs. There should be no customer facing impact any more.

The issue appears to be related to the setup of our caching backend. We will temporarily disable transparent caching to prevent setup of GitHub Actions from failing. This will not affect customers that are using our useblacksmith/cache or useblacksmith/setup-* actions.