Status | Blacksmith - Notice history

All systems operational

Blacksmith Managed Runners - Operational

100% - uptime
Jul 2024 · 99.85%Aug · 100.0%Sep · 99.95%

Previous page

Next page

Jul 2024
Aug 2024
Sep 2024

API - Operational

100% - uptime
Jul 2024 · 100.0%Aug · 100.0%Sep · 100.0%

Previous page

Next page

Jul 2024
Aug 2024
Sep 2024

Website - Operational

100% - uptime
Jul 2024 · 100.0%Aug · 100.0%Sep · 100.0%

Previous page

Next page

Jul 2024
Aug 2024
Sep 2024

Github → Actions - Operational

Github → API Requests - Operational

Github → Webhooks - Operational

Notice history

Sep 2024

Increased job queue times
  • Resolved
    Resolved
    This incident has been resolved.
  • Investigating
    Investigating

    We're seeing GitHub Action jobs take longer than usual to get assigned to our runners. We are currently investigating this incident.

actions/checkout failing with upstream service unavailable
  • Resolved
    Resolved

    This is now resolved.

  • Monitoring
    Monitoring

    We're no longer seeing issues with checkout actions and GitHub has closed their incident: https://www.githubstatus.com/incidents/1g9v7rry4z86. We're still monitoring this issue.

  • Identified
    Identified

    GitHub has started an incident on their end: https://www.githubstatus.com/incidents/1g9v7rry4z86. We're still monitoring this issue on our end.

  • Investigating
    Investigating

    Customers are reporting checkout actions failing with Error: Name or service not known (internal-api.service.iad.github.net:443. We are currently investigating this incident.

Github `actions/checkout` is failing with `fatal: protocol error: bad pack header` when trying to pull the repository
  • Resolved
    Resolved

    This incident has been resolved by GitHub, operations should be back to normal.

  • Identified
    Identified

    GitHub has opened an incident, we will continue monitoring the status of this - https://www.githubstatus.com/incidents/q3xqwmcxzkqq

  • Investigating
    Investigating

    Both GitHub hosted runners and Blacksmith hosted runners are seeing an error when trying to checkout the repository in a GitHub workflow. This is indicative of an outage or bad deploy on GitHub's end, we are monitoring the situation.

Degraded performance with Actions
  • Resolved
    Resolved
    This incident has been resolved.
  • Investigating
    Investigating

    We are noticing that GitHub is not processing certain job requests, like canceling a job, and it could lead to increased queuing. We will update here as we find more information.

Flakiness in `ubuntu/22.04/prod jammy/main arm64` mirror causing apt installs to occasionally fail
  • Resolved
    Resolved

    This incident has been resolved. This has been fixed upstream.

  • Investigating
    Investigating

    We are noticing some tests failing while setting up the job and installing the required dependencies with the error File has unexpected size (43334 != 43043). Mirror sync in progress?

    There is an open issue tracking the problem with the upstream mirror https://github.com/microsoft/linux-package-repositories/issues/174, we will continue to monitor the situation.

Aug 2024

Delays in job pickup times
  • Resolved
    Resolved
    This incident has been resolved.
  • Investigating
    Investigating

    We are currently investigating this incident, we're noticing a slight up tick in time taken to adopt jobs.

Delays in job pickups
  • Resolved
    Resolved

    This incident is now resolved.

  • Investigating
    Investigating

    We're seeing delays in getting jobs assigned to our runners from GitHub. We are currently investigating this incident.

GitHub reported degraded performance for Actions
GitHub is reporting degraded performance with their GitHub Actions service
  • Resolved
    Resolved

    GitHub has resolved the incident, jobs are operating normally.

  • Investigating
    Investigating

    We are currently investigating this incident being reported by GitHub https://www.githubstatus.com/incidents/mrpx4trfk45z. Jobs are currently getting picked up as normal.

Major GitHub outage
  • Resolved
    Resolved

    This incident has been marked resolved by GitHub, jobs are running on Blacksmith normally again. We will continue to monitor any fallout from this outage.

  • Identified
    Identified

    GitHub is rolling back its database infrastructure change. They said they are seeing service health improvements and monitoring for a full recovery. https://www.githubstatus.com/incidents/kz4khcgdsfdv

  • Investigating
    Investigating

    GitHub has reported a major outage https://www.githubstatus.com/incidents/kz4khcgdsfdv

Jul 2024

Delay in Actions job pickup
  • Resolved
    Resolved
    This incident has been resolved.
  • Monitoring
    Monitoring

    We're seeing some delays in GitHub handing our runners jobs in the aftermath of the incident earlier today. We've deployed some mitigation steps to help drain the backlog of jobs. We're monitoring recovery.

GitHub reported a partial outage of their Actions service
  • Update
    Update
    This incident has been resolved.
  • Resolved
    Resolved

    GitHub reported a partial outage of their GitHub Actions service which was causing some jobs to not get picked up - https://www.githubstatus.com/incidents/dy0rbtrxtwlh.

    The incident has now been declared resolved.

Missed webhook deliveries from GitHub
  • Resolved
    Resolved

    Resolved.

GitHub is taking longer to assign jobs to our runners
  • Resolved
    Resolved
    This incident has been resolved.
  • Update
    Update

    GitHub has reverted the bad rollout as per this comment: https://github.com/actions/runner/issues/3381#issuecomment-2253186060. We're continuing to monitor.

  • Monitoring
    Monitoring

    This incident is linked to a bad rollout of a new runner version by GitHub: https://github.com/actions/runner/issues/3381. We're monitoring the situation.

  • Investigating
    Investigating
    We are currently investigating this incident.
GitHub is reporting delays in webhooks which could cause delays in job pickup times
  • Resolved
    Resolved
    This incident has been resolved.
  • Monitoring
    Monitoring

    GitHub has reported an incident declaring delays across GitHub services. We are monitoring the situation - https://www.githubstatus.com/incidents/n2969465tx0b

Jul 2024 to Sep 2024

Next