blacksmith - Increased queue times in the US region – Incident details

This incident has been resolved, queue times are back to normal. The longer queue times were a combination of an outage with our upstream DB provider and a need for us to scale up our fleet to absorb higher than normal traffic. We are working with our DB provider to prevent such an outage in the future, and have scaled up our fleet to prevent such saturation in the future.

We sincerely apologize for the prolonged queueing that our customers saw today. We take such a prolonged outage very seriously and will work on hardening our systems based on our discoveries from today. If you still see any queued jobs we would recommend canceling and re-triggering jobs to see normal adoption.

Queue times are stabilizing and we're continuing to monitor.

We are monitoring increased queue times in relation to an incident with our database provider and are working towards a fix

We are continuing to monitor increased queue times.

We are continuing to monitor increasing queue times. Our team is working on mitigations to bring times back down to normal.

Increased queue times continue but we are seeing improvement and are monitoring the situation.

We are continuing to investigate increased queue times in the US and are working to resolve this.

We are continuing to investigate increased queue times in the US