Status | Blacksmith - Notice history

All systems operational

Blacksmith Managed Runners - Operational

100% - uptime
Oct 2024 · 99.99%Nov · 100.0%Dec · 99.98%

Previous page

Next page

Oct 2024
Nov 2024
Dec 2024

API - Operational

100% - uptime
Oct 2024 · 99.99%Nov · 100.0%Dec · 100.0%

Previous page

Next page

Oct 2024
Nov 2024
Dec 2024

Website - Operational

100% - uptime
Oct 2024 · 100.0%Nov · 100.0%Dec · 100.0%

Previous page

Next page

Oct 2024
Nov 2024
Dec 2024

Github → Actions - Operational

Github → API Requests - Operational

Github → Webhooks - Operational

Notice history

Dec 2024

Increased error rate in our control plane affecting dashboard login
  • Resolved
    Resolved

    This incident has been resolved, the increased error rates were because of a scheduled maintenance of our database provider.

  • Investigating
    Investigating

    We have been notified of a spike in error rates coming from our Blacksmith control plane. Users may be unable to login to their dashboard.

Jobs hanging on `git checkout` step in GitHub workflows
  • Resolved
    Resolved

    This incident has been resolved, our US runners are back to having normal download times. This was caused due to an upstream networking flake with our data centre provider.

  • Investigating
    Investigating

    We have been alerted of some instances of the git checkout step hanging on runners in our US fleet. We are looking into the issue and will post an update once we have understood the cause.

Increased queue times in jobs being assigned to our runner
  • Resolved
    Resolved

    All services are working as expected.

  • Monitoring
    Monitoring

    Blacksmith runners are adopting jobs normally now. A network misconfiguration resulted in some of our services seeing degraded performance. We are continuing to monitor the recovery of our website and dashboard.

  • Identified
    Identified

    We have identified the issue that is preventing our runners from adopting jobs. We have deployed a fix and expect complete recovery shortly.

  • Investigating
    Investigating

    GitHub is not assigning jobs to some of our runners, we are monitoring the situation.

Increased queue times in jobs being assigned to our runner
  • Resolved
    Resolved

    Queue times have returned back to normal.

  • Investigating
    Investigating

    We are noticing signs of slightly elevated queue times. We are monitoring the situation and will post an update once they return to normal.

Degradation in GitHub Services
  • Resolved
    Resolved

    GitHub has marked the incident as resolved.

  • Identified
    Identified

    GitHub as declared an incident - https://www.githubstatus.com/incidents/4349zxvb8stp. We are monitoring the situation

  • Investigating
    Investigating

    We are noticing signs of an outage on GitHub's end. This is affecting their Actions service and is resulting in no jobs being assigned to our runners. Their website and pull requests also appear to be affected.

Nov 2024

Increased job queue times from GitHub
  • Resolved
    Resolved
    This incident has been resolved.
  • Monitoring
    Monitoring

    We are noticing partial degradation in GitHub Action services. This is manifesting as longer queue times once again.

  • Resolved
    Resolved

    This incident has been resolved, queue times are returning to normal. We will continue monitoring the situation.

  • Investigating
    Investigating

    We are noticing partial degradation in GitHub Action services. This is manifesting as longer queue times and the logs not streaming through for some jobs.

Increased job queue times from GitHub
  • Resolved
    Resolved

    Queue times have returned back to normal, we will continue monitoring the situation.

  • Investigating
    Investigating

    We are noticing GitHub taking longer than usual to assign jobs to our runners, we are looking into this at the moment.

Increased job queue times from GitHub
  • Resolved
    Resolved
    This incident has been resolved.
  • Investigating
    Investigating

    We're seeing GitHub take longer than usual to assign jobs to our runners. We are currently investigating this incident.

Increased job queue times from GitHub
  • Resolved
    Resolved
    This incident has been resolved.
  • Monitoring
    Monitoring

    We are seeing some signs of recovery again and are closely monitoring the situation.

  • Identified
    Identified

    We're seeing signs of the same incident reported earlier today, with GitHub's control plane not assigning jobs to our runners.

Increased job queue times from GitHub
  • Resolved
    Resolved

    The incident is now resolved.

  • Monitoring
    Monitoring

    We're seeing some signs of recovery as jobs are starting to get assigned to our runners as expected.

  • Identified
    Identified

    We've validated that this seems to be at least a partial outage on GitHub's side. We're closely monitoring the situation.

Oct 2024

Increased queue times because of GitHub not assigning jobs to our runners
  • Resolved
    Resolved

    GitHub has resolved the incident on their end, we are seeing normal queue times once again.

  • Update
    Update

    GitHub is still continuing to investigate delays to status updates to Actions Workflow Runs, Workflow Job Runs, and Check Steps.

  • Monitoring
    Monitoring

    GitHub has declared an incident, we are continuing to monitor the situation - https://www.githubstatus.com/incidents/9yk1fbk0qjjc

  • Investigating
    Investigating

    We are currently noticing higher than normal queue times because of GitHub not assigning jobs to some of our runners. We are noticing excessive queueing on ubuntu-latest as well.

GitHub is reporting server errors and not assigning jobs to our runners
  • Resolved
    Resolved

    This incident has been resolved, error rates are back to zero and jobs are getting assigned to our runners.

  • Investigating
    Investigating

    We are currently investigating this incident. It appears to be an outage on GitHub's endpoints and webhook delivery.

Connection issues with our database provider.
  • Resolved
    Resolved
    This incident has been resolved.
  • Monitoring
    Monitoring

    We're seeing some recovery and are monitoring the situation.

  • Investigating
    Investigating

    We're seeing intermittent connection failures to our database provider. We are currently investigating this incident.

Connection issues with our database provider
  • Resolved
    Resolved
    This incident has been resolved.
  • Monitoring
    Monitoring

    The connection issues have dissipated, we are monitoring the system to ensure normal queue times.

  • Investigating
    Investigating

    We're seeing some connection issues with our database provider that may result in GitHub Action cache failures. We are investigating the issue.

Longer queue times for ARM runners
  • Resolved
    Resolved
    This incident has been resolved.
  • Monitoring
    Monitoring

    We've deployed a fix and are seeing short queue times again. We're actively monitoring this fix.

  • Investigating
    Investigating

    We are looking into reported delays in job pickup by our ARM runners.

Oct 2024 to Dec 2024

Next