Status | Blacksmith - Small % of missed webhooks from GitHub resulting in runners getting queued for longer – Incident details

All systems operational

Small % of missed webhooks from GitHub resulting in runners getting queued for longer

Resolved
Major outage
Started 6 days agoLasted about 1 hour
Updates
  • Resolved
    Resolved

    This incident has been resolved, we have added additional monitoring so that if this happens again we are able to investigate more effectively.

  • Monitoring
    Monitoring

    We have re-deployed certain resources and that has restored webhook delivery rates to normal. We will continue to investigate what happened here but customers should no longer be seeing extended queue times.

  • Investigating
    Investigating

    We are currently investigating this incident. We are noticing a small percent of queued web hooks not reaching our control plane. This is resulting in customer's seeing elevated queue times for some of their jobs.