Status | Blacksmith - Degradation in queue times causing runners to sporadically not get assigned a job – Incident details

All systems operational

Degradation in queue times causing runners to sporadically not get assigned a job

Resolved
Degraded performance
Started 3 months agoLasted about 6 hours

Affected

Blacksmith Managed Runners

Degraded performance from 10:38 AM to 3:32 PM, Operational from 3:32 PM to 4:39 PM

Updates
  • Resolved
    Resolved

    This incident has been resolved, we have requested our database provider for a postmortem of the upstream incident.

  • Monitoring
    Monitoring

    Our database provider has rolled out a fix and is claiming recovery. We are also seeing signs of recovery on our end.

  • Identified
    Identified

    Our database provider has declared an incident that is directly affecting our control plane - https://status.supabase.com/incidents/kfrmjq5z9n8x. We are continuing to work with them on a timeline for a fix.

  • Update
    Update
    We are continuing to investigate the cause of this incident. All signs point to a degradation in our upstream database provider. We are actively working with them to resolve the incident.
  • Investigating
    Investigating

    The deployed fix has not completely mitigated the outage. Customers may also be noticing degraded load times for app.blacksmith.sh

  • Identified
    Identified

    We have identified the issue as likely being our database connection pool limits, we are rolling out a fix to see if it helps with the situation.

  • Investigating
    Investigating
    We are currently investigating this incident.