Status | Blacksmith - Notice history

All systems operational

Blacksmith Managed Runners - Operational

100% - uptime
Mar 2024 · 100.0%Apr · 100.0%May · 99.52%

Previous page

Next page

Mar 2024
Apr 2024
May 2024

API - Operational

100% - uptime
Mar 2024 · 100.0%Apr · 100.0%May · 99.89%

Previous page

Next page

Mar 2024
Apr 2024
May 2024

Website - Operational

100% - uptime
Mar 2024 · 99.92%Apr · 100.0%May · 99.89%

Previous page

Next page

Mar 2024
Apr 2024
May 2024

Github → Actions - Operational

Github → API Requests - Operational

Github → Webhooks - Operational

Notice history

May 2024

Longer queue times for GitHub jobs
  • Resolved
    Resolved
    This incident has been resolved.
  • Identified
    Identified

    We've identified the cause of the degradation being some stuck jobs not freeing up CPUs, we're working on a patch to bring queue times back to normal.

  • Investigating
    Investigating

    Users may be noticing longer queue times for GitHub jobs, we are investigating the cause of this right now.

Increased GitHub job queue times
  • Resolved
    Resolved
    This incident has been resolved.
  • Monitoring
    Monitoring

    GitHub is reporting recovery of the GitHub Actions service, we are also observing most jobs being successfully picked up by our Blacksmith runners.

  • Update
    Update

    GitHub has opened an official incident to monitor this outage - https://www.githubstatus.com/incidents/nj47vccwm2zj

  • Identified
    Identified

    This appears to be a GitHub outage, neither GitHub hosted runners nor Blacksmith hosted runners are being handed jobs. We are continuing to monitor the situation.

  • Investigating
    Investigating

    We are noticing an increase in job queue times because of a delay in GitHub handing jobs to our runners

GitHub not assigning jobs to runners
  • Resolved
    Resolved
    This incident has been resolved.
  • Identified
    Identified

    The issue appears to be on GitHub’s end, they are not assigning jobs to any hosted runners. We will continue to monitor the situation and relay updates.

  • Investigating
    Investigating

    We are currently investigating this incident. We are observing GitHub not assigning jobs to hosted runners.

GitHub Action Runners outage
  • Resolved
    Resolved

    GitHub Action Runners is now operational! This update was created by an automated monitoring service.

  • Investigating
    Investigating

    GitHub Action Runners cannot be accessed at the moment. This incident was created by an automated monitoring service.

API outage
  • Resolved
    Resolved

    API is now operational! This update was created by an automated monitoring service.

  • Investigating
    Investigating

    API cannot be accessed at the moment. This incident was created by an automated monitoring service.

Apr 2024

Delay in picking up jobs because of ongoing GitHub outage
  • Resolved
    Resolved
    This incident has been resolved.
  • Investigating
    Investigating

    Customers may notice intermittent delays in job pickup times because of an ongoing GitHub outage affecting "Actions, API Requests, Issues, Webhooks, Pull Requests and Git Operations" - https://www.githubstatus.com/incidents/7txcg0j03kkg

    We are actively monitoring this situation.

Sporadic connection timeouts from our API endpoints
  • Resolved
    Resolved
    This incident has been resolved.
  • Investigating
    Investigating

    We were notified of sporadic failures coming from some of our API endpoints. This was attributed to degraded service from our hosting provider fly.io. Our system was completely operational through this partial outage and job adoption and execution were unaffected.

Mar 2024

Degraded service of runner queue
  • Resolved
    Resolved
    This incident has been resolved.
  • Investigating
    Investigating

    Our connection attempts to the runner queue was temporarily timing out. Some jobs may have been in the queued state for longer than expected.

Degraded performance of GitHub Actions due to a GitHub outage
  • Resolved
    Resolved
    This incident has been resolved.
  • Investigating
    Investigating

    GitHub is reporting degraded delivery of webhooks. Please refer to https://www.githubstatus.com/incidents/wcl1sw4mzg60. We will update this status once GitHub marks the incident as resolved.

    GitHub has resolved the outage and our services should be running normally now. If you run into an issue please reach out to hello@blacksmith.sh

Mar 2024 to May 2024

Next