blacksmith - Notice history

All systems operational

100% - uptime

eu-central Storage Cluster - Operational

100% - uptime
Mar 2025 · 100.0%Apr · 100.0%May · 100.0%

Previous page

Next page

Mar 2025
Apr 2025
May 2025

us-west Storage Cluster - Operational

100% - uptime
Mar 2025 · 100.0%Apr · 100.0%May · 100.0%

Previous page

Next page

Mar 2025
Apr 2025
May 2025

eu-west Storage Cluster - Operational

100% - uptime
Mar 2025 · 100.0%Apr · 100.0%May · 100.0%

Previous page

Next page

Mar 2025
Apr 2025
May 2025

API - Operational

100% - uptime
Mar 2025 · 100.0%Apr · 99.81%May · 100.0%

Previous page

Next page

Mar 2025
Apr 2025
May 2025
100% - uptime

https://blacksmith.sh - Operational

100% - uptime
Mar 2025 · 100.0%Apr · 100.0%May · 100.0%

Previous page

Next page

Mar 2025
Apr 2025
May 2025

Github → Actions - Operational

Github → API Requests - Operational

Github → Webhooks - Operational

Codesmith - Operational

100% - uptime
Mar 2025 · 100.0%Apr · 100.0%May · 100.0%

Previous page

Next page

Mar 2025
Apr 2025
May 2025

Dashboard - Operational

100% - uptime
Mar 2025 · 100.0%Apr · 100.0%May · 100.0%

Previous page

Next page

Mar 2025
Apr 2025
May 2025

Notice history

May 2025

Receiving reports of cache failures in Blacksmith runners
  • Resolved
    UTC
    Resolved

    GitHub has resolved the incident, we will continue to monitor interactions with our cache.

  • Identified
    UTC
    Identified

    GitHub have declared an incident related to the upstream cache service - https://www.githubstatus.com/incidents/9hzy25gws8vh

  • Update
    UTC
    Update

    An upstream issue has been filed against the actions/cache repository indicating other users are also noticing errors of the form The runner was not able to contact the cache service - https://github.com/actions/cache/issues/1611

    We are continuing to investigate the root cause.

  • Investigating
    UTC
    Investigating

    We are currently investigating this incident, we are receiving reports of failures to contact the Blacksmith cache.

GitHub Actions jobs losing connection to the GitHub control plane
  • Resolved
    UTC
    Resolved

    The bug was in the part of our system that allocates static IPs for our runners, but had a wider blast radius that was causing other VMs to get terminated pre-emptively -- the fix has been rolled to the fleet and we're monitoring the incident.

  • Investigating
    UTC
    Investigating

    We're seeing some number of GitHub actions jobs lose their connection to the GitHub control plane. We are currently investigating this incident.

Increased queueing of GitHub runners in our US region
  • Resolved
    UTC
    Resolved

    This incident has been resolved, we ran into capacity limits in our US fleet that have since been addressed. We'd like to apologize for the disruption this may have caused to your workflows.

  • Investigating
    UTC
    Investigating

    We are currently getting alerted of increased queue times in our US region, we are investigating the issue.

Reports of GitHub Actions never ending even though the steps have all completed
  • Resolved
    UTC
    Resolved

    We are no longer receiving reports of this behaviour, we will continue to monitor the situation.

  • Investigating
    UTC
    Investigating

    We are receiving reports from several customer where they see their GitHub Actions completing all the steps, but then spinning in the final Complete job stage. This is causing jobs to hit their timeouts and eventually fail. There have been numerous reports of this in the past:
    https://github.com/actions/runner/issues/3462
    https://github.com/actions/runner/issues/2409

    GitHub has not yet opened an incident but we expect there to be one soon.

Degraded interaction of our ARM runners with the shared Docker layer cache
  • Resolved
    UTC
    Resolved

    This incident has been resolved, we had a bad host machine in our fleet that has since been decommissioned.

  • Investigating
    UTC
    Investigating
    We are currently investigating this incident.

Apr 2025

Noticing degradation in GitHub API interactions like repository checkout and logging in to GHCR
  • Resolved
    UTC
    Resolved

    This incident has been resolved, GitHub API interactions appear to be back to normal.

  • Investigating
    UTC
    Investigating

    We are currently investigating this incident. We have been alerted about some steps in GitHub Action workflows such as checking out the repository and logging in to GHCR taking much longer than usual.

Dashboard analytics are stale
  • Resolved
    UTC
    Resolved
    This incident has been resolved.
  • Identified
    UTC
    Identified

    Our dashboard analytics are stale due to the previous incident. We've identified the issue and are working on a fix.

Blacksmith's control plane is unresponsive
  • Resolved
    UTC
    Resolved

    This incident has been resolved.

  • Monitoring
    UTC
    Monitoring

    We've deployed a fix and are monitoring the incident.

  • Identified
    UTC
    Identified

    We've run into an issue with our upstream database provider. We're working on mitigating this incident.

  • Investigating
    UTC
    Investigating
    We are currently investigating this incident.
Increased queue times in the US region
  • Resolved
    UTC
    Resolved

    This incident has been resolved, queue times are back to normal in the US region.

  • Investigating
    UTC
    Investigating

    We are currently noticing higher than normal queue times for orgs running in our US region, we are looking into whether this is an upstream GitHub issue

GitHub outage preventing runners from getting assigned to Blacksmith
  • Resolved
    UTC
    Resolved
    This incident has been resolved.
  • Monitoring
    UTC
    Monitoring

    We are seeing signs of recovery in GitHub assigning jobs to our VMs, we will continue to monitor queue times.

  • Investigating
    UTC
    Investigating

    We are getting notified of a potential GitHub outage that is causing jobs to not get assigned to our runners, we are monitoring the situation.

Mar 2025

Degraded performance of Docker builders in the US region
  • Resolved
    UTC
    Resolved

    The upstream network degradation has been resolved and Docker builds are functioning as normal.

  • Monitoring
    UTC
    Monitoring

    We received information from an upstream provider about degradation in their network stack. We are continuing to monitor recovery of our services.

  • Investigating
    UTC
    Investigating

    We are currently investigating reports of degraded performance of our Docker builders in the US region. This may affect jobs using useblackmith/build-push-action@v1

High rate of failures in the Blacksmith Cache
  • Resolved
    UTC
    Resolved

    This incident has been resolved, cache operations have returned to normal.

  • Identified
    UTC
    Identified

    We've identified a fix and are working to deploy it.

  • Investigating
    UTC
    Investigating

    We are currently investigating a higher than normal rate of failures when hitting our Blacksmith cace.

Increased queue times for GitHub Actions
  • Resolved
    UTC
    Resolved
    This incident has been resolved.
  • Investigating
    UTC
    Investigating

    GitHub is taking longer than usual to assign jobs to our runners which may manifest as longer queue times. We are looking into this incident.

Degraded performance of GitHub Actions reported by Github
  • Resolved
    UTC
    Resolved

    GitHub has marked the incident as resolved.

  • Investigating
    UTC
    Investigating

    GitHub has opened an incident declaring degradation in their GitHub Actions service - https://www.githubstatus.com/incidents/lg4s05t6ttxb. We are monitoring the impact of this on Blacksmith runners.

Delays in Actions jobs getting picked up
  • Resolved
    UTC
    Resolved

    This incident has been resolved, queue times have returned back to normal.

  • Investigating
    UTC
    Investigating

    We are currently investigating delays in GitHub Actions jobs being picked up by our runners in the EU.

Mar 2025 to May 2025

Next