Status | Blacksmith - Increased queue times in the US region – Incident details

Increased queue times in the US region

Resolved
Degraded performance
Started 1 day agoLasted about 7 hours

Affected

Blacksmith Managed Runners

Degraded performance from 5:50 PM to 11:48 PM, Operational from 11:48 PM to 12:31 AM

Updates
  • Resolved
    Resolved

    This incident has been resolved, queue times are back to normal. The longer queue times were a combination of an outage with our upstream DB provider and a need for us to scale up our fleet to absorb higher than normal traffic. We are working with our DB provider to prevent such an outage in the future, and have scaled up our fleet to prevent such saturation in the future.

    We sincerely apologize for the prolonged queueing that our customers saw today. We take such a prolonged outage very seriously and will work on hardening our systems based on our discoveries from today. If you still see any queued jobs we would recommend canceling and re-triggering jobs to see normal adoption.

  • Update
    Update

    Queue times are stabilizing and we're continuing to monitor.

  • Update
    Update

    We are monitoring increased queue times in relation to an incident with our database provider and are working towards a fix

  • Update
    Update

    We are continuing to monitor increased queue times.

  • Update
    Update

    We are continuing to monitor increasing queue times. Our team is working on mitigations to bring times back down to normal.

  • Monitoring
    Monitoring

    Increased queue times continue but we are seeing improvement and are monitoring the situation.

  • Update
    Update

    We are continuing to investigate increased queue times in the US and are working to resolve this.

  • Update
    Update

    We are continuing to investigate increased queue times in the US

  • Investigating
    Investigating

    We are being notified of extended queue times in the US region, and are looking into this.