blacksmith - Increased queue times in the US region – Incident details

Increased queue times in the US region

ResolvedDegraded performance6 hours 41 minutes

Affected

Blacksmith Managed Runners

  • ResolvedUTC

    This incident has been resolved, queue times are back to normal. The longer queue times were a combination of an outage with our upstream DB provider and a need for us to scale up our fleet to absorb higher than normal traffic. We are working with our DB provider to prevent such an outage in the future, and have scaled up our fleet to prevent such saturation in the future.

    We sincerely apologize for the prolonged queueing that our customers saw today. We take such a prolonged outage very seriously and will work on hardening our systems based on our discoveries from today. If you still see any queued jobs we would recommend canceling and re-triggering jobs to see normal adoption.

  • UpdateUTC

    Queue times are stabilizing and we're continuing to monitor.

  • UpdateUTC

    We are monitoring increased queue times in relation to an incident with our database provider and are working towards a fix

  • UpdateUTC

    We are continuing to monitor increased queue times.

  • UpdateUTC

    We are continuing to monitor increasing queue times. Our team is working on mitigations to bring times back down to normal.

  • MonitoringUTC

    Increased queue times continue but we are seeing improvement and are monitoring the situation.

  • UpdateUTC

    We are continuing to investigate increased queue times in the US and are working to resolve this.

  • UpdateUTC

    We are continuing to investigate increased queue times in the US

  • InvestigatingUTC

    We are being notified of extended queue times in the US region, and are looking into this.