Status | Blacksmith - Notice history

All systems operational

Blacksmith Managed Runners - Operational

100% - uptime
Nov 2024 · 100.0%Dec · 99.98%Jan 2025 · 100.0%

Previous page

Next page

Nov 2024
Dec 2024
Jan 2025

Incremental Docker Builders - Operational

100% - uptime
Nov 2024 · 100.0%Dec · 100.0%Jan 2025 · 100.0%

Previous page

Next page

Nov 2024
Dec 2024
Jan 2025

API - Operational

100% - uptime
Nov 2024 · 100.0%Dec · 100.0%Jan 2025 · 100.0%

Previous page

Next page

Nov 2024
Dec 2024
Jan 2025

Website - Operational

100% - uptime
Nov 2024 · 100.0%Dec · 100.0%Jan 2025 · 100.0%

Previous page

Next page

Nov 2024
Dec 2024
Jan 2025

Github → Actions - Operational

Github → API Requests - Operational

Github → Webhooks - Operational

Notice history

Jan 2025

Increased time for GitHub to assign jobs to our runners
  • Resolved
    Resolved

    GitHub has resolved the incident, queue times are returning to normal on Blacksmith runners.

  • Investigating
    Investigating

    GitHub has declared an incident which is causing some degradation in their ability to assign jobs to our runners - https://www.githubstatus.com/incidents/nm83zrdky73y. Customer may notice increased queue times during this period.

Reports of job setup failing due to network degradation
  • Resolved
    Resolved

    This incident has been resolved, runners are consistently working as expected.

  • Monitoring
    Monitoring

    We are seeing signs of recovery, jobs are consistently being able to setup their environments and pull artifacts. We are continuing to monitor the situation until all our monitoring returns to its baseline.

  • Update
    Update

    We are seeing degradation across both our US and EU fleets, which are on different networking stacks. This suggests that there is a degradation on GitHub's end, as can be seen on downdetector - https://downdetector.com/status/github/. We're still investigating on our end.

  • Investigating
    Investigating

    We are seeing reports of some network degradation across our fleet. It is unclear whether this is on GitHub's end or not. We are looking into the issue.

Increased queue times for GitHub Actions
  • Resolved
    Resolved

    This incident has been resolved, queue times are back to normal.

  • Investigating
    Investigating

    We are noticing increased queued times in GitHub assigning jobs to our runners. We are monitoring the situation.

Increased queue times for GitHub Actions
  • Resolved
    Resolved
    This incident has been resolved.
  • Investigating
    Investigating

    Customers may notice increased queue times of their GitHub Actions. We are monitoring the situation as GitHub is taking longer to assign jobs to our runners.

GitHub reporting degradation in Git operations
  • Resolved
    Resolved
    This incident has been resolved.
  • Monitoring
    Monitoring

    GitHub posted an update

    "We've identified a cause of degraded git operations, which may affect other GitHub services that rely upon git. We're working to remediate."

  • Identified
    Identified

    GitHub posted an update "Actions is experiencing degraded performance. We are continuing to investigate."

    https://www.githubstatus.com/incidents/qd96yfgvmcf9

    We're continuing to monitor this.

  • Investigating
    Investigating

    GitHub has opened an incident reporting degradation in Git operations. This may manifest as checkouts failing in GitHub Action workflows - https://www.githubstatus.com/incidents/qd96yfgvmcf9

Dec 2024

Increased error rate in our control plane affecting dashboard login
  • Resolved
    Resolved

    This incident has been resolved, the increased error rates were because of a scheduled maintenance of our database provider.

  • Investigating
    Investigating

    We have been notified of a spike in error rates coming from our Blacksmith control plane. Users may be unable to login to their dashboard.

Jobs hanging on `git checkout` step in GitHub workflows
  • Resolved
    Resolved

    This incident has been resolved, our US runners are back to having normal download times. This was caused due to an upstream networking flake with our data centre provider.

  • Investigating
    Investigating

    We have been alerted of some instances of the git checkout step hanging on runners in our US fleet. We are looking into the issue and will post an update once we have understood the cause.

Increased queue times in jobs being assigned to our runner
  • Resolved
    Resolved

    All services are working as expected.

  • Monitoring
    Monitoring

    Blacksmith runners are adopting jobs normally now. A network misconfiguration resulted in some of our services seeing degraded performance. We are continuing to monitor the recovery of our website and dashboard.

  • Identified
    Identified

    We have identified the issue that is preventing our runners from adopting jobs. We have deployed a fix and expect complete recovery shortly.

  • Investigating
    Investigating

    GitHub is not assigning jobs to some of our runners, we are monitoring the situation.

Increased queue times in jobs being assigned to our runner
  • Resolved
    Resolved

    Queue times have returned back to normal.

  • Investigating
    Investigating

    We are noticing signs of slightly elevated queue times. We are monitoring the situation and will post an update once they return to normal.

Degradation in GitHub Services
  • Resolved
    Resolved

    GitHub has marked the incident as resolved.

  • Identified
    Identified

    GitHub as declared an incident - https://www.githubstatus.com/incidents/4349zxvb8stp. We are monitoring the situation

  • Investigating
    Investigating

    We are noticing signs of an outage on GitHub's end. This is affecting their Actions service and is resulting in no jobs being assigned to our runners. Their website and pull requests also appear to be affected.

Nov 2024

Increased job queue times from GitHub
  • Resolved
    Resolved
    This incident has been resolved.
  • Monitoring
    Monitoring

    We are noticing partial degradation in GitHub Action services. This is manifesting as longer queue times once again.

  • Resolved
    Resolved

    This incident has been resolved, queue times are returning to normal. We will continue monitoring the situation.

  • Investigating
    Investigating

    We are noticing partial degradation in GitHub Action services. This is manifesting as longer queue times and the logs not streaming through for some jobs.

Increased job queue times from GitHub
  • Resolved
    Resolved

    Queue times have returned back to normal, we will continue monitoring the situation.

  • Investigating
    Investigating

    We are noticing GitHub taking longer than usual to assign jobs to our runners, we are looking into this at the moment.

Increased job queue times from GitHub
  • Resolved
    Resolved
    This incident has been resolved.
  • Investigating
    Investigating

    We're seeing GitHub take longer than usual to assign jobs to our runners. We are currently investigating this incident.

Increased job queue times from GitHub
  • Resolved
    Resolved
    This incident has been resolved.
  • Monitoring
    Monitoring

    We are seeing some signs of recovery again and are closely monitoring the situation.

  • Identified
    Identified

    We're seeing signs of the same incident reported earlier today, with GitHub's control plane not assigning jobs to our runners.

Increased job queue times from GitHub
  • Resolved
    Resolved

    The incident is now resolved.

  • Monitoring
    Monitoring

    We're seeing some signs of recovery as jobs are starting to get assigned to our runners as expected.

  • Identified
    Identified

    We've validated that this seems to be at least a partial outage on GitHub's side. We're closely monitoring the situation.

Nov 2024 to Jan 2025

Next