Status | Blacksmith - Notice history

All systems operational

100% - uptime

eu-central Storage Cluster - Operational

100% - uptime
Apr 2025 · 100.0%May · 100.0%Jun · 100.0%

Previous page

Next page

Apr 2025
May 2025
Jun 2025

us-west Storage Cluster - Operational

100% - uptime
Apr 2025 · 100.0%May · 100.0%Jun · 100.0%

Previous page

Next page

Apr 2025
May 2025
Jun 2025

eu-central x86 - Operational

100% - uptime
Apr 2025 · 100.0%May · 100.0%Jun · 100.0%

Previous page

Next page

Apr 2025
May 2025
Jun 2025

us-west ARM - Operational

100% - uptime
Apr 2025 · 100.0%May · 100.0%Jun · 100.0%

Previous page

Next page

Apr 2025
May 2025
Jun 2025

us-west x86 - Operational

100% - uptime
Apr 2025 · 100.0%May · 100.0%Jun · 100.0%

Previous page

Next page

Apr 2025
May 2025
Jun 2025

Actions Cache - Operational

100% - uptime
Apr 2025 · 100.0%May · 100.0%Jun · 100.0%

Previous page

Next page

Apr 2025
May 2025
Jun 2025

eu-west x86 - Operational

100% - uptime
Apr 2025 · 100.0%May · 100.0%Jun · 100.0%

Previous page

Next page

Apr 2025
May 2025
Jun 2025

API - Operational

100% - uptime
Apr 2025 · 99.81%May · 100.0%Jun · 100.0%

Previous page

Next page

Apr 2025
May 2025
Jun 2025

us-central MacOS - Operational

100% - uptime
Apr 2025 · 100.0%May · 100.0%Jun · 100.0%

Previous page

Next page

Apr 2025
May 2025
Jun 2025

Website - Operational

100% - uptime
Apr 2025 · 100.0%May · 100.0%Jun · 100.0%

Previous page

Next page

Apr 2025
May 2025
Jun 2025

Github → Actions - Operational

Github → API Requests - Operational

Github → Webhooks - Operational

Notice history

Jun 2025

Landing page undergoing maintenance
  • Completed
    June 30, 2025 at 10:16 PM
    Completed
    June 30, 2025 at 10:16 PM
    Maintenance has completed successfully.
  • In progress
    June 30, 2025 at 9:30 PM
    In progress
    June 30, 2025 at 9:30 PM

    There is ongoing maintenance on our landing page (blacksmith.sh), but our dashboard and support portals are unaffected https://app.blacksmith.sh/

Seeing timeouts in downloading some third-party actions in GitHub Actions
  • Resolved
    Resolved

    This incident has been resolved, our customers are no longer seeing these slowdowns or errors when downloading third party actions.

  • Investigating
    Investigating

    Customers are reporting timeouts in downloading some actions during job runs. This implies an issue upstream with GitHub, we are continuing to monitor the situation.

Potential GitHub outage preventing jobs from being assigned to our runners
  • Resolved
    Resolved

    GitHub have marked their incident as closed, our runners are back to behaving as expected.

  • Update
    Update

    GitHub has declared a widespread incident that is affecting Actions as well, we will continue to monitor their status page for updates - https://www.githubstatus.com/incidents/y7lb2rg4btd7

  • Investigating
    Investigating

    We are seeing an uptick in upstream errors from the GitHub control plane, preventing runners from being assigned jobs. We are monitoring the situation.

Increased queue times for runners
  • Resolved
    Resolved
    This incident has been resolved.
  • Investigating
    Investigating

    We are noticing increased queue times for runners in our EU region. We are currently investigating this incident.

Degraded performance of our incremental docker builds in our EU region
  • Resolved
    Resolved

    This incident has been resolved, the fix has been deployed to our storage cluster and builds in the EU region are working as expected.

  • Monitoring
    Monitoring

    We have deployed a fix and are monitoring the cluster as it comes back up.

  • Investigating
    Investigating
    We are currently investigating this incident.

May 2025

Receiving reports of cache failures in Blacksmith runners
  • Resolved
    Resolved

    GitHub has resolved the incident, we will continue to monitor interactions with our cache.

  • Identified
    Identified

    GitHub have declared an incident related to the upstream cache service - https://www.githubstatus.com/incidents/9hzy25gws8vh

  • Update
    Update

    An upstream issue has been filed against the actions/cache repository indicating other users are also noticing errors of the form The runner was not able to contact the cache service - https://github.com/actions/cache/issues/1611

    We are continuing to investigate the root cause.

  • Investigating
    Investigating

    We are currently investigating this incident, we are receiving reports of failures to contact the Blacksmith cache.

GitHub Actions jobs losing connection to the GitHub control plane
  • Resolved
    Resolved

    The bug was in the part of our system that allocates static IPs for our runners, but had a wider blast radius that was causing other VMs to get terminated pre-emptively -- the fix has been rolled to the fleet and we're monitoring the incident.

  • Investigating
    Investigating

    We're seeing some number of GitHub actions jobs lose their connection to the GitHub control plane. We are currently investigating this incident.

Increased queueing of GitHub runners in our US region
  • Resolved
    Resolved

    This incident has been resolved, we ran into capacity limits in our US fleet that have since been addressed. We'd like to apologize for the disruption this may have caused to your workflows.

  • Investigating
    Investigating

    We are currently getting alerted of increased queue times in our US region, we are investigating the issue.

Reports of GitHub Actions never ending even though the steps have all completed
  • Resolved
    Resolved

    We are no longer receiving reports of this behaviour, we will continue to monitor the situation.

  • Investigating
    Investigating

    We are receiving reports from several customer where they see their GitHub Actions completing all the steps, but then spinning in the final Complete job stage. This is causing jobs to hit their timeouts and eventually fail. There have been numerous reports of this in the past:
    https://github.com/actions/runner/issues/3462
    https://github.com/actions/runner/issues/2409

    GitHub has not yet opened an incident but we expect there to be one soon.

Degraded interaction of our ARM runners with the shared Docker layer cache
  • Resolved
    Resolved

    This incident has been resolved, we had a bad host machine in our fleet that has since been decommissioned.

  • Investigating
    Investigating
    We are currently investigating this incident.

Apr 2025

Noticing degradation in GitHub API interactions like repository checkout and logging in to GHCR
  • Resolved
    Resolved

    This incident has been resolved, GitHub API interactions appear to be back to normal.

  • Investigating
    Investigating

    We are currently investigating this incident. We have been alerted about some steps in GitHub Action workflows such as checking out the repository and logging in to GHCR taking much longer than usual.

Dashboard analytics are stale
  • Resolved
    Resolved
    This incident has been resolved.
  • Identified
    Identified

    Our dashboard analytics are stale due to the previous incident. We've identified the issue and are working on a fix.

Blacksmith's control plane is unresponsive
  • Resolved
    Resolved

    This incident has been resolved.

  • Monitoring
    Monitoring

    We've deployed a fix and are monitoring the incident.

  • Identified
    Identified

    We've run into an issue with our upstream database provider. We're working on mitigating this incident.

  • Investigating
    Investigating
    We are currently investigating this incident.
Increased queue times in the US region
  • Resolved
    Resolved

    This incident has been resolved, queue times are back to normal in the US region.

  • Investigating
    Investigating

    We are currently noticing higher than normal queue times for orgs running in our US region, we are looking into whether this is an upstream GitHub issue

GitHub outage preventing runners from getting assigned to Blacksmith
  • Resolved
    Resolved
    This incident has been resolved.
  • Monitoring
    Monitoring

    We are seeing signs of recovery in GitHub assigning jobs to our VMs, we will continue to monitor queue times.

  • Investigating
    Investigating

    We are getting notified of a potential GitHub outage that is causing jobs to not get assigned to our runners, we are monitoring the situation.

Apr 2025 to Jun 2025

Next