Increased queue times in the US region
Affected
- Resolved · UTC
This incident has been resolved, queue times are back to normal. The longer queue times were a combination of an outage with our upstream DB provider and a need for us to scale up our fleet to absorb higher than normal traffic. We are working with our DB provider to prevent such an outage in the future, and have scaled up our fleet to prevent such saturation in the future.
We sincerely apologize for the prolonged queueing that our customers saw today. We take such a prolonged outage very seriously and will work on hardening our systems based on our discoveries from today. If you still see any queued jobs we would recommend canceling and re-triggering jobs to see normal adoption. - Update · UTC
Queue times are stabilizing and we're continuing to monitor.
- Update · UTC
We are monitoring increased queue times in relation to an incident with our database provider and are working towards a fix
- Update · UTC
We are continuing to monitor increased queue times.
- Update · UTC
We are continuing to monitor increasing queue times. Our team is working on mitigations to bring times back down to normal.
- Monitoring · UTC
Increased queue times continue but we are seeing improvement and are monitoring the situation.
- Update · UTC
We are continuing to investigate increased queue times in the US and are working to resolve this.
- Update · UTC
We are continuing to investigate increased queue times in the US
- Investigating · UTC
We are being notified of extended queue times in the US region, and are looking into this.
