Status | Blacksmith - Notice history

All systems operational

Blacksmith Managed Runners - Operational

100% - uptime
Mar 2025 · 100.0%Apr · 99.81%May · 100.0%

Previous page

Next page

Mar 2025
Apr 2025
May 2025

Incremental Docker Builders - Operational

100% - uptime
Mar 2025 · 100.0%Apr · 99.81%May · 100.0%

Previous page

Next page

Mar 2025
Apr 2025
May 2025

API - Operational

100% - uptime
Mar 2025 · 100.0%Apr · 99.81%May · 100.0%

Previous page

Next page

Mar 2025
Apr 2025
May 2025

Website - Operational

100% - uptime
Mar 2025 · 100.0%Apr · 100.0%May · 100.0%

Previous page

Next page

Mar 2025
Apr 2025
May 2025

Github → Actions - Operational

Github → API Requests - Operational

Github → Webhooks - Operational

Notice history

May 2025

Reports of GitHub Actions never ending even though the steps have all completed
  • Resolved
    Resolved

    We are no longer receiving reports of this behaviour, we will continue to monitor the situation.

  • Investigating
    Investigating

    We are receiving reports from several customer where they see their GitHub Actions completing all the steps, but then spinning in the final Complete job stage. This is causing jobs to hit their timeouts and eventually fail. There have been numerous reports of this in the past:
    https://github.com/actions/runner/issues/3462
    https://github.com/actions/runner/issues/2409

    GitHub has not yet opened an incident but we expect there to be one soon.

Degraded interaction of our ARM runners with the shared Docker layer cache
  • Resolved
    Resolved

    This incident has been resolved, we had a bad host machine in our fleet that has since been decommissioned.

  • Investigating
    Investigating
    We are currently investigating this incident.
GitHub reporting degradation in their Actions services
  • Resolved
    Resolved
    This incident has been resolved.
  • Monitoring
    Monitoring

    GitHub have reported an incident with their Actions service. We don't see any impact on Blacksmith at the moment but we are monitoring the situation - https://www.githubstatus.com/incidents/42gtccf6dd98

Degradation in Blacksmith cache performance
  • Resolved
    Resolved

    This incident has been resolved, cache services are back to normal.

  • Monitoring
    Monitoring

    The network degradation is showing signs of recovery and we're noticing normal performance across our cache services.

  • Update
    Update

    We are continuing to look into the degraded performance of our runners in the EU region. We are being alerted of some upstream degradation in our network stack.

  • Investigating
    Investigating

    We are currently investigating this incident, we are getting reports of slow cache uploads and downloads in our EU region.

Degradation in Blacksmith Cache service
  • Resolved
    Resolved

    This incident has been resolved, cache services are back to normal.

  • Investigating
    Investigating

    We are currently investigating this incident, we are being alerted of a degradation in our cache service.

Apr 2025

Noticing degradation in GitHub API interactions like repository checkout and logging in to GHCR
  • Resolved
    Resolved

    This incident has been resolved, GitHub API interactions appear to be back to normal.

  • Investigating
    Investigating

    We are currently investigating this incident. We have been alerted about some steps in GitHub Action workflows such as checking out the repository and logging in to GHCR taking much longer than usual.

Dashboard analytics are stale
  • Resolved
    Resolved
    This incident has been resolved.
  • Identified
    Identified

    Our dashboard analytics are stale due to the previous incident. We've identified the issue and are working on a fix.

Blacksmith's control plane is unresponsive
  • Resolved
    Resolved

    This incident has been resolved.

  • Monitoring
    Monitoring

    We've deployed a fix and are monitoring the incident.

  • Identified
    Identified

    We've run into an issue with our upstream database provider. We're working on mitigating this incident.

  • Investigating
    Investigating
    We are currently investigating this incident.
Increased queue times in the US region
  • Resolved
    Resolved

    This incident has been resolved, queue times are back to normal in the US region.

  • Investigating
    Investigating

    We are currently noticing higher than normal queue times for orgs running in our US region, we are looking into whether this is an upstream GitHub issue

GitHub outage preventing runners from getting assigned to Blacksmith
  • Resolved
    Resolved
    This incident has been resolved.
  • Monitoring
    Monitoring

    We are seeing signs of recovery in GitHub assigning jobs to our VMs, we will continue to monitor queue times.

  • Investigating
    Investigating

    We are getting notified of a potential GitHub outage that is causing jobs to not get assigned to our runners, we are monitoring the situation.

Mar 2025

Degraded performance of Docker builders in the US region
  • Resolved
    Resolved

    The upstream network degradation has been resolved and Docker builds are functioning as normal.

  • Monitoring
    Monitoring

    We received information from an upstream provider about degradation in their network stack. We are continuing to monitor recovery of our services.

  • Investigating
    Investigating

    We are currently investigating reports of degraded performance of our Docker builders in the US region. This may affect jobs using useblackmith/build-push-action@v1

High rate of failures in the Blacksmith Cache
  • Resolved
    Resolved

    This incident has been resolved, cache operations have returned to normal.

  • Identified
    Identified

    We've identified a fix and are working to deploy it.

  • Investigating
    Investigating

    We are currently investigating a higher than normal rate of failures when hitting our Blacksmith cace.

Increased queue times for GitHub Actions
  • Resolved
    Resolved
    This incident has been resolved.
  • Investigating
    Investigating

    GitHub is taking longer than usual to assign jobs to our runners which may manifest as longer queue times. We are looking into this incident.

Degraded performance of GitHub Actions reported by Github
  • Resolved
    Resolved

    GitHub has marked the incident as resolved.

  • Investigating
    Investigating

    GitHub has opened an incident declaring degradation in their GitHub Actions service - https://www.githubstatus.com/incidents/lg4s05t6ttxb. We are monitoring the impact of this on Blacksmith runners.

Delays in Actions jobs getting picked up
  • Resolved
    Resolved

    This incident has been resolved, queue times have returned back to normal.

  • Investigating
    Investigating

    We are currently investigating delays in GitHub Actions jobs being picked up by our runners in the EU.

Mar 2025 to May 2025

Next