blacksmith - Notice history

All systems operational

100% - uptime

eu-central Storage Cluster - Operational

100% - uptime
Dec 2024 · 100.0%Jan 2025 · 100.0%Feb · 100.0%

Previous page

Next page

Dec 2024
Jan 2025
Feb 2025

us-west Storage Cluster - Operational

100% - uptime
Dec 2024 · 100.0%Jan 2025 · 100.0%Feb · 100.0%

Previous page

Next page

Dec 2024
Jan 2025
Feb 2025

eu-west Storage Cluster - Operational

100% - uptime
Dec 2024 · 100.0%Jan 2025 · 100.0%Feb · 100.0%

Previous page

Next page

Dec 2024
Jan 2025
Feb 2025

API - Operational

100% - uptime
Dec 2024 · 100.0%Jan 2025 · 100.0%Feb · 100.0%

Previous page

Next page

Dec 2024
Jan 2025
Feb 2025
100% - uptime

https://blacksmith.sh - Operational

100% - uptime
Dec 2024 · 100.0%Jan 2025 · 100.0%Feb · 100.0%

Previous page

Next page

Dec 2024
Jan 2025
Feb 2025

Github → Actions - Operational

Github → API Requests - Operational

Github → Webhooks - Operational

Codesmith - Operational

100% - uptime
Dec 2024 · 100.0%Jan 2025 · 100.0%Feb · 100.0%

Previous page

Next page

Dec 2024
Jan 2025
Feb 2025

Dashboard - Operational

100% - uptime
Dec 2024 · 100.0%Jan 2025 · 100.0%Feb · 100.0%

Previous page

Next page

Dec 2024
Jan 2025
Feb 2025

Notice history

Feb 2025

Incident affecting GitHub Actions
  • Resolved
    UTC
    Resolved
    This incident has been resolved.
  • Investigating
    UTC
    Investigating

    GitHub has declared an incident affecting Actions and Packages: https://www.githubstatus.com/incidents/2lxm4wb8wy3r. We're investigating the impact of this incident on our services.

Increased failure rate for Blacksmith Cache interactions
  • Resolved
    UTC
    Resolved
    This incident has been resolved.
  • Monitoring
    UTC
    Monitoring

    We're seeing signs of recovery after deploying a few fixes, we are continuing to monitor our cache health.

Increased failure rate for Blacksmith Cache interactions
  • Resolved
    UTC
    Resolved
    This incident has been resolved.
  • Monitoring
    UTC
    Monitoring

    We're seeing signs of recovery and are monitoring the issue.

  • Investigating
    UTC
    Investigating

    We're seeing a higher-than-usual rate of failures for cache restores from the Blacksmith cache. We are currently investigating this incident.

Increased time for GitHub to assign jobs to our runners
  • Resolved
    UTC
    Resolved
    This incident has been resolved.
  • Investigating
    UTC
    Investigating

    We are being alerted about some orgs seeing higher than usual queue times. GitHub is not assigning jobs to our runners in a timely fashion, we are looking into this.

Increased queue times on US runners
  • Resolved
    UTC
    Resolved

    This incident has been resolved, US queue times are back to normal.

  • Investigating
    UTC
    Investigating

    We are noticing jobs taking longer than usual to get picked up in our US fleet.

Jan 2025

Increased time for GitHub to assign jobs to our runners
  • Resolved
    UTC
    Resolved

    GitHub has resolved the incident, queue times are returning to normal on Blacksmith runners.

  • Investigating
    UTC
    Investigating

    GitHub has declared an incident which is causing some degradation in their ability to assign jobs to our runners - https://www.githubstatus.com/incidents/nm83zrdky73y. Customer may notice increased queue times during this period.

Reports of job setup failing due to network degradation
  • Resolved
    UTC
    Resolved

    This incident has been resolved, runners are consistently working as expected.

  • Monitoring
    UTC
    Monitoring

    We are seeing signs of recovery, jobs are consistently being able to setup their environments and pull artifacts. We are continuing to monitor the situation until all our monitoring returns to its baseline.

  • Update
    UTC
    Update

    We are seeing degradation across both our US and EU fleets, which are on different networking stacks. This suggests that there is a degradation on GitHub's end, as can be seen on downdetector - https://downdetector.com/status/github/. We're still investigating on our end.

  • Investigating
    UTC
    Investigating

    We are seeing reports of some network degradation across our fleet. It is unclear whether this is on GitHub's end or not. We are looking into the issue.

Increased queue times for GitHub Actions
  • Resolved
    UTC
    Resolved

    This incident has been resolved, queue times are back to normal.

  • Investigating
    UTC
    Investigating

    We are noticing increased queued times in GitHub assigning jobs to our runners. We are monitoring the situation.

Increased queue times for GitHub Actions
  • Resolved
    UTC
    Resolved
    This incident has been resolved.
  • Investigating
    UTC
    Investigating

    Customers may notice increased queue times of their GitHub Actions. We are monitoring the situation as GitHub is taking longer to assign jobs to our runners.

GitHub reporting degradation in Git operations
  • Resolved
    UTC
    Resolved
    This incident has been resolved.
  • Monitoring
    UTC
    Monitoring

    GitHub posted an update

    "We've identified a cause of degraded git operations, which may affect other GitHub services that rely upon git. We're working to remediate."

  • Identified
    UTC
    Identified

    GitHub posted an update "Actions is experiencing degraded performance. We are continuing to investigate."

    https://www.githubstatus.com/incidents/qd96yfgvmcf9

    We're continuing to monitor this.

  • Investigating
    UTC
    Investigating

    GitHub has opened an incident reporting degradation in Git operations. This may manifest as checkouts failing in GitHub Action workflows - https://www.githubstatus.com/incidents/qd96yfgvmcf9

Dec 2024

Increased error rate in our control plane affecting dashboard login
  • Resolved
    UTC
    Resolved

    This incident has been resolved, the increased error rates were because of a scheduled maintenance of our database provider.

  • Investigating
    UTC
    Investigating

    We have been notified of a spike in error rates coming from our Blacksmith control plane. Users may be unable to login to their dashboard.

Jobs hanging on `git checkout` step in GitHub workflows
  • Resolved
    UTC
    Resolved

    This incident has been resolved, our US runners are back to having normal download times. This was caused due to an upstream networking flake with our data centre provider.

  • Investigating
    UTC
    Investigating

    We have been alerted of some instances of the git checkout step hanging on runners in our US fleet. We are looking into the issue and will post an update once we have understood the cause.

Increased queue times in jobs being assigned to our runner
  • Resolved
    UTC
    Resolved

    All services are working as expected.

  • Monitoring
    UTC
    Monitoring

    Blacksmith runners are adopting jobs normally now. A network misconfiguration resulted in some of our services seeing degraded performance. We are continuing to monitor the recovery of our website and dashboard.

  • Identified
    UTC
    Identified

    We have identified the issue that is preventing our runners from adopting jobs. We have deployed a fix and expect complete recovery shortly.

  • Investigating
    UTC
    Investigating

    GitHub is not assigning jobs to some of our runners, we are monitoring the situation.

Increased queue times in jobs being assigned to our runner
  • Resolved
    UTC
    Resolved

    Queue times have returned back to normal.

  • Investigating
    UTC
    Investigating

    We are noticing signs of slightly elevated queue times. We are monitoring the situation and will post an update once they return to normal.

Degradation in GitHub Services
  • Resolved
    UTC
    Resolved

    GitHub has marked the incident as resolved.

  • Identified
    UTC
    Identified

    GitHub as declared an incident - https://www.githubstatus.com/incidents/4349zxvb8stp. We are monitoring the situation

  • Investigating
    UTC
    Investigating

    We are noticing signs of an outage on GitHub's end. This is affecting their Actions service and is resulting in no jobs being assigned to our runners. Their website and pull requests also appear to be affected.

Dec 2024 to Feb 2025

Next