Status | Blacksmith - Notice history

All systems operational

Blacksmith Managed Runners - Operational

100% - uptime
Feb 2025 · 100.0%Mar · 100.0%Apr · 100.0%

Previous page

Next page

Feb 2025
Mar 2025
Apr 2025

Incremental Docker Builders - Operational

100% - uptime
Feb 2025 · 100.0%Mar · 100.0%Apr · 100.0%

Previous page

Next page

Feb 2025
Mar 2025
Apr 2025

API - Operational

100% - uptime
Feb 2025 · 100.0%Mar · 100.0%Apr · 100.0%

Previous page

Next page

Feb 2025
Mar 2025
Apr 2025

Website - Operational

100% - uptime
Feb 2025 · 100.0%Mar · 100.0%Apr · 100.0%

Previous page

Next page

Feb 2025
Mar 2025
Apr 2025

Github → Actions - Operational

Github → API Requests - Operational

Github → Webhooks - Operational

Notice history

Apr 2025

GitHub outage preventing runners from getting assigned to Blacksmith
  • Resolved
    Resolved
    This incident has been resolved.
  • Monitoring
    Monitoring

    We are seeing signs of recovery in GitHub assigning jobs to our VMs, we will continue to monitor queue times.

  • Investigating
    Investigating

    We are getting notified of a potential GitHub outage that is causing jobs to not get assigned to our runners, we are monitoring the situation.

Network degradation for runners in our US region
  • Resolved
    Resolved

    This incident has been resolved, network health is back to normal in our US fleet.

  • Monitoring
    Monitoring

    We have been informed that the upstream networking issue has now been resolved and service should be returning to normal. We are working with our vendor to better understand how to avoid this in the future.

  • Update
    Update

    Our vendor is actively working on a fix for this network degradation. In the mean time we are removing all the affected servers from our fleet to help bring service back to normal.

  • Identified
    Identified

    We are specifically seeing packets being dropped when downloading from api.github.com. We continue to look into the source of the degradation.

  • Investigating
    Investigating

    We are currently investigating reports from an upstream vendor of a degraded network stack in our US region. This could manifest as hanging GitHub Actions runners or slow/stuck steps in the workflow.

Mar 2025

Degraded performance of Docker builders in the US region
  • Resolved
    Resolved

    The upstream network degradation has been resolved and Docker builds are functioning as normal.

  • Monitoring
    Monitoring

    We received information from an upstream provider about degradation in their network stack. We are continuing to monitor recovery of our services.

  • Investigating
    Investigating

    We are currently investigating reports of degraded performance of our Docker builders in the US region. This may affect jobs using useblackmith/build-push-action@v1

High rate of failures in the Blacksmith Cache
  • Resolved
    Resolved

    This incident has been resolved, cache operations have returned to normal.

  • Identified
    Identified

    We've identified a fix and are working to deploy it.

  • Investigating
    Investigating

    We are currently investigating a higher than normal rate of failures when hitting our Blacksmith cace.

Increased queue times for GitHub Actions
  • Resolved
    Resolved
    This incident has been resolved.
  • Investigating
    Investigating

    GitHub is taking longer than usual to assign jobs to our runners which may manifest as longer queue times. We are looking into this incident.

Degraded performance of GitHub Actions reported by Github
  • Resolved
    Resolved

    GitHub has marked the incident as resolved.

  • Investigating
    Investigating

    GitHub has opened an incident declaring degradation in their GitHub Actions service - https://www.githubstatus.com/incidents/lg4s05t6ttxb. We are monitoring the impact of this on Blacksmith runners.

Delays in Actions jobs getting picked up
  • Resolved
    Resolved

    This incident has been resolved, queue times have returned back to normal.

  • Investigating
    Investigating

    We are currently investigating delays in GitHub Actions jobs being picked up by our runners in the EU.

Feb 2025

Incident affecting GitHub Actions
  • Resolved
    Resolved
    This incident has been resolved.
  • Investigating
    Investigating

    GitHub has declared an incident affecting Actions and Packages: https://www.githubstatus.com/incidents/2lxm4wb8wy3r. We're investigating the impact of this incident on our services.

Increased failure rate for Blacksmith Cache interactions
  • Resolved
    Resolved
    This incident has been resolved.
  • Monitoring
    Monitoring

    We're seeing signs of recovery after deploying a few fixes, we are continuing to monitor our cache health.

Increased failure rate for Blacksmith Cache interactions
  • Resolved
    Resolved
    This incident has been resolved.
  • Monitoring
    Monitoring

    We're seeing signs of recovery and are monitoring the issue.

  • Investigating
    Investigating

    We're seeing a higher-than-usual rate of failures for cache restores from the Blacksmith cache. We are currently investigating this incident.

Increased time for GitHub to assign jobs to our runners
  • Resolved
    Resolved
    This incident has been resolved.
  • Investigating
    Investigating

    We are being alerted about some orgs seeing higher than usual queue times. GitHub is not assigning jobs to our runners in a timely fashion, we are looking into this.

Increased queue times on US runners
  • Resolved
    Resolved

    This incident has been resolved, US queue times are back to normal.

  • Investigating
    Investigating

    We are noticing jobs taking longer than usual to get picked up in our US fleet.

Feb 2025 to Apr 2025

Next