Status | Blacksmith - Notice history

All systems operational

Blacksmith Managed Runners - Operational

100% - uptime
Sep 2024 · 100%Oct · 99.95%Nov · 99.99%

Previous page

Next page

Sep 2024
Oct 2024
Nov 2024

API - Operational

100% - uptime
Sep 2024 · 100%Oct · 100.0%Nov · 99.99%

Previous page

Next page

Sep 2024
Oct 2024
Nov 2024

Website - Operational

100% - uptime
Sep 2024 · 100%Oct · 100.0%Nov · 100.0%

Previous page

Next page

Sep 2024
Oct 2024
Nov 2024

Github → Actions - Operational

Github → API Requests - Operational

Github → Webhooks - Operational

Notice history

Nov 2024

Increased job queue times from GitHub
  • Resolved
    Resolved
    This incident has been resolved.
  • Investigating
    Investigating

    We're seeing GitHub take longer than usual to assign jobs to our runners. We are currently investigating this incident.

Increased job queue times from GitHub
  • Resolved
    Resolved
    This incident has been resolved.
  • Monitoring
    Monitoring

    We are seeing some signs of recovery again and are closely monitoring the situation.

  • Identified
    Identified

    We're seeing signs of the same incident reported earlier today, with GitHub's control plane not assigning jobs to our runners.

Increased job queue times from GitHub
  • Resolved
    Resolved

    The incident is now resolved.

  • Monitoring
    Monitoring

    We're seeing some signs of recovery as jobs are starting to get assigned to our runners as expected.

  • Identified
    Identified

    We've validated that this seems to be at least a partial outage on GitHub's side. We're closely monitoring the situation.

Increased job queue times
  • Resolved
    Resolved

    This incident has been resolved, job queue times are back to normal.

  • Monitoring
    Monitoring

    We are currently noticing higher than normal job queue times for some orgs on Blacksmith. We are monitoring this and looking into potential causes.

Oct 2024

Increased queue times because of GitHub not assigning jobs to our runners
  • Resolved
    Resolved

    GitHub has resolved the incident on their end, we are seeing normal queue times once again.

  • Update
    Update

    GitHub is still continuing to investigate delays to status updates to Actions Workflow Runs, Workflow Job Runs, and Check Steps.

  • Monitoring
    Monitoring

    GitHub has declared an incident, we are continuing to monitor the situation - https://www.githubstatus.com/incidents/9yk1fbk0qjjc

  • Investigating
    Investigating

    We are currently noticing higher than normal queue times because of GitHub not assigning jobs to some of our runners. We are noticing excessive queueing on ubuntu-latest as well.

GitHub is reporting server errors and not assigning jobs to our runners
  • Resolved
    Resolved

    This incident has been resolved, error rates are back to zero and jobs are getting assigned to our runners.

  • Investigating
    Investigating

    We are currently investigating this incident. It appears to be an outage on GitHub's endpoints and webhook delivery.

Connection issues with our database provider.
  • Resolved
    Resolved
    This incident has been resolved.
  • Monitoring
    Monitoring

    We're seeing some recovery and are monitoring the situation.

  • Investigating
    Investigating

    We're seeing intermittent connection failures to our database provider. We are currently investigating this incident.

Connection issues with our database provider
  • Resolved
    Resolved
    This incident has been resolved.
  • Monitoring
    Monitoring

    The connection issues have dissipated, we are monitoring the system to ensure normal queue times.

  • Investigating
    Investigating

    We're seeing some connection issues with our database provider that may result in GitHub Action cache failures. We are investigating the issue.

Longer queue times for ARM runners
  • Resolved
    Resolved
    This incident has been resolved.
  • Monitoring
    Monitoring

    We've deployed a fix and are seeing short queue times again. We're actively monitoring this fix.

  • Investigating
    Investigating

    We are looking into reported delays in job pickup by our ARM runners.

Sep 2024

Increased job queue times
  • Resolved
    Resolved
    This incident has been resolved.
  • Investigating
    Investigating

    We're seeing GitHub Action jobs take longer than usual to get assigned to our runners. We are currently investigating this incident.

actions/checkout failing with upstream service unavailable
  • Resolved
    Resolved

    This is now resolved.

  • Monitoring
    Monitoring

    We're no longer seeing issues with checkout actions and GitHub has closed their incident: https://www.githubstatus.com/incidents/1g9v7rry4z86. We're still monitoring this issue.

  • Identified
    Identified

    GitHub has started an incident on their end: https://www.githubstatus.com/incidents/1g9v7rry4z86. We're still monitoring this issue on our end.

  • Investigating
    Investigating

    Customers are reporting checkout actions failing with Error: Name or service not known (internal-api.service.iad.github.net:443. We are currently investigating this incident.

Github `actions/checkout` is failing with `fatal: protocol error: bad pack header` when trying to pull the repository
  • Resolved
    Resolved

    This incident has been resolved by GitHub, operations should be back to normal.

  • Identified
    Identified

    GitHub has opened an incident, we will continue monitoring the status of this - https://www.githubstatus.com/incidents/q3xqwmcxzkqq

  • Investigating
    Investigating

    Both GitHub hosted runners and Blacksmith hosted runners are seeing an error when trying to checkout the repository in a GitHub workflow. This is indicative of an outage or bad deploy on GitHub's end, we are monitoring the situation.

Degraded performance with Actions
  • Resolved
    Resolved
    This incident has been resolved.
  • Investigating
    Investigating

    We are noticing that GitHub is not processing certain job requests, like canceling a job, and it could lead to increased queuing. We will update here as we find more information.

Flakiness in `ubuntu/22.04/prod jammy/main arm64` mirror causing apt installs to occasionally fail
  • Resolved
    Resolved

    This incident has been resolved. This has been fixed upstream.

  • Investigating
    Investigating

    We are noticing some tests failing while setting up the job and installing the required dependencies with the error File has unexpected size (43334 != 43043). Mirror sync in progress?

    There is an open issue tracking the problem with the upstream mirror https://github.com/microsoft/linux-package-repositories/issues/174, we will continue to monitor the situation.

Sep 2024 to Nov 2024

Next