Status | Blacksmith - Notice history

All systems operational

Blacksmith Managed Runners - Operational

100% - uptime
Nov 2024 · 100.0%Dec · 99.98%Jan 2025 · 100.0%

Previous page

Next page

Nov 2024
Dec 2024
Jan 2025

API - Operational

100% - uptime
Nov 2024 · 100.0%Dec · 100.0%Jan 2025 · 100.0%

Previous page

Next page

Nov 2024
Dec 2024
Jan 2025

Website - Operational

100% - uptime
Nov 2024 · 100.0%Dec · 100.0%Jan 2025 · 100.0%

Previous page

Next page

Nov 2024
Dec 2024
Jan 2025

Github → Actions - Operational

Github → API Requests - Operational

Github → Webhooks - Operational

Notice history

Jan 2025

Increased queue times for GitHub Actions
  • Resolved
    Resolved
    This incident has been resolved.
  • Investigating
    Investigating

    Customers may notice increased queue times of their GitHub Actions. We are monitoring the situation as GitHub is taking longer to assign jobs to our runners.

GitHub reporting degradation in Git operations
  • Resolved
    Resolved
    This incident has been resolved.
  • Monitoring
    Monitoring

    GitHub posted an update

    "We've identified a cause of degraded git operations, which may affect other GitHub services that rely upon git. We're working to remediate."

  • Identified
    Identified

    GitHub posted an update "Actions is experiencing degraded performance. We are continuing to investigate."

    https://www.githubstatus.com/incidents/qd96yfgvmcf9

    We're continuing to monitor this.

  • Investigating
    Investigating

    GitHub has opened an incident reporting degradation in Git operations. This may manifest as checkouts failing in GitHub Action workflows - https://www.githubstatus.com/incidents/qd96yfgvmcf9

Delays in jobs being assigned to Blacksmith runners
  • Update
    Update
    This incident has been resolved.
  • Resolved
    Resolved

    We're seeing some delays with GitHub assigning jobs to our runners. We're investigating the situation.

Incident with Actions
  • Resolved
    Resolved
    This incident has been resolved.
  • Update
    Update

    GitHub has expanded the scope of their reported incident to indicate a much more widespread outage: https://www.githubstatus.com/incidents/plgzz71xn6zq

  • Investigating
    Investigating

    GitHub has opened an incident stating that Actions is experiencing degraded performance: https://www.githubstatus.com/incidents/plgzz71xn6zq. We're investigating the impact of this incident on our services.

Delays in jobs being assigned to our runners, and dropped webhooks
  • Resolved
    Resolved
    This incident has been resolved.
  • Monitoring
    Monitoring

    Update from GitHub:

    "Webhooks is experiencing degraded performance. We are continuing to investigate."

    "We have identified a configuration issue that we believe is the source of the Action workflow job delays and page latency increases. We are continuing to investigate and mitigate the issue."

  • Identified
    Identified

    GitHub has declared an incident of their own: https://www.githubstatus.com/incidents/dk61qxd21mtl

  • Investigating
    Investigating

    We're noticing a significantly higher-than-usual delay in jobs being assigned to our runners, and are seeing a small percentage of webhook notifications get completely dropped.

Dec 2024

Increased error rate in our control plane affecting dashboard login
  • Resolved
    Resolved

    This incident has been resolved, the increased error rates were because of a scheduled maintenance of our database provider.

  • Investigating
    Investigating

    We have been notified of a spike in error rates coming from our Blacksmith control plane. Users may be unable to login to their dashboard.

Jobs hanging on `git checkout` step in GitHub workflows
  • Resolved
    Resolved

    This incident has been resolved, our US runners are back to having normal download times. This was caused due to an upstream networking flake with our data centre provider.

  • Investigating
    Investigating

    We have been alerted of some instances of the git checkout step hanging on runners in our US fleet. We are looking into the issue and will post an update once we have understood the cause.

Increased queue times in jobs being assigned to our runner
  • Resolved
    Resolved

    All services are working as expected.

  • Monitoring
    Monitoring

    Blacksmith runners are adopting jobs normally now. A network misconfiguration resulted in some of our services seeing degraded performance. We are continuing to monitor the recovery of our website and dashboard.

  • Identified
    Identified

    We have identified the issue that is preventing our runners from adopting jobs. We have deployed a fix and expect complete recovery shortly.

  • Investigating
    Investigating

    GitHub is not assigning jobs to some of our runners, we are monitoring the situation.

Increased queue times in jobs being assigned to our runner
  • Resolved
    Resolved

    Queue times have returned back to normal.

  • Investigating
    Investigating

    We are noticing signs of slightly elevated queue times. We are monitoring the situation and will post an update once they return to normal.

Degradation in GitHub Services
  • Resolved
    Resolved

    GitHub has marked the incident as resolved.

  • Identified
    Identified

    GitHub as declared an incident - https://www.githubstatus.com/incidents/4349zxvb8stp. We are monitoring the situation

  • Investigating
    Investigating

    We are noticing signs of an outage on GitHub's end. This is affecting their Actions service and is resulting in no jobs being assigned to our runners. Their website and pull requests also appear to be affected.

Nov 2024

Increased job queue times from GitHub
  • Resolved
    Resolved
    This incident has been resolved.
  • Monitoring
    Monitoring

    We are noticing partial degradation in GitHub Action services. This is manifesting as longer queue times once again.

  • Resolved
    Resolved

    This incident has been resolved, queue times are returning to normal. We will continue monitoring the situation.

  • Investigating
    Investigating

    We are noticing partial degradation in GitHub Action services. This is manifesting as longer queue times and the logs not streaming through for some jobs.

Increased job queue times from GitHub
  • Resolved
    Resolved

    Queue times have returned back to normal, we will continue monitoring the situation.

  • Investigating
    Investigating

    We are noticing GitHub taking longer than usual to assign jobs to our runners, we are looking into this at the moment.

Increased job queue times from GitHub
  • Resolved
    Resolved
    This incident has been resolved.
  • Investigating
    Investigating

    We're seeing GitHub take longer than usual to assign jobs to our runners. We are currently investigating this incident.

Increased job queue times from GitHub
  • Resolved
    Resolved
    This incident has been resolved.
  • Monitoring
    Monitoring

    We are seeing some signs of recovery again and are closely monitoring the situation.

  • Identified
    Identified

    We're seeing signs of the same incident reported earlier today, with GitHub's control plane not assigning jobs to our runners.

Increased job queue times from GitHub
  • Resolved
    Resolved

    The incident is now resolved.

  • Monitoring
    Monitoring

    We're seeing some signs of recovery as jobs are starting to get assigned to our runners as expected.

  • Identified
    Identified

    We've validated that this seems to be at least a partial outage on GitHub's side. We're closely monitoring the situation.

Nov 2024 to Jan 2025

Next