blacksmith - Notice history

All systems operational

100% - uptime

eu-central Storage Cluster - Operational

100% - uptime
May 2024 · 100.0%Jun · 100.0%Jul · 100.0%

Previous page

Next page

May 2024
Jun 2024
Jul 2024

us-west Storage Cluster - Operational

100% - uptime
May 2024 · 100.0%Jun · 100.0%Jul · 100.0%

Previous page

Next page

May 2024
Jun 2024
Jul 2024

eu-west Storage Cluster - Operational

100% - uptime
May 2024 · 100.0%Jun · 100.0%Jul · 100.0%

Previous page

Next page

May 2024
Jun 2024
Jul 2024

us-east Storage Cluster - Operational

100% - uptime
May 2024 · 100.0%Jun · 100.0%Jul · 100.0%

Previous page

Next page

May 2024
Jun 2024
Jul 2024

API - Operational

100% - uptime
May 2024 · 99.89%Jun · 100.0%Jul · 100.0%

Previous page

Next page

May 2024
Jun 2024
Jul 2024
100% - uptime

https://blacksmith.sh - Operational

100% - uptime
May 2024 · 100.0%Jun · 100.0%Jul · 100.0%

Previous page

Next page

May 2024
Jun 2024
Jul 2024

Github → Actions - Operational

Github → API Requests - Operational

Github → Webhooks - Operational

Codesmith - Operational

100% - uptime
May 2024 · 100.0%Jun · 100.0%Jul · 100.0%

Previous page

Next page

May 2024
Jun 2024
Jul 2024

Dashboard - Operational

100% - uptime
May 2024 · 100.0%Jun · 100.0%Jul · 100.0%

Previous page

Next page

May 2024
Jun 2024
Jul 2024

Notice history

View current status

Jul 2024

Delay in Actions job pickup
ResolvedDegraded performance33 minutes
  • Resolved
    UTC
    Resolved
    This incident has been resolved.
  • Monitoring
    UTC
    Monitoring

    We're seeing some delays in GitHub handing our runners jobs in the aftermath of the incident earlier today. We've deployed some mitigation steps to help drain the backlog of jobs. We're monitoring recovery.

GitHub reported a partial outage of their Actions service
ResolvedPartial outage
  • Update
    UTC
    Update
    This incident has been resolved.
  • Resolved
    UTC
    Resolved

    GitHub reported a partial outage of their GitHub Actions service which was causing some jobs to not get picked up - https://www.githubstatus.com/incidents/dy0rbtrxtwlh.

    The incident has now been declared resolved.

Missed webhook deliveries from GitHub
Resolved
  • Resolved
    UTC
    Resolved

    Resolved.

GitHub is taking longer to assign jobs to our runners
Resolved1 hour
  • Resolved
    UTC
    Resolved
    This incident has been resolved.
  • Update
    UTC
    Update

    GitHub has reverted the bad rollout as per this comment: https://github.com/actions/runner/issues/3381#issuecomment-2253186060. We're continuing to monitor.

  • Monitoring
    UTC
    Monitoring

    This incident is linked to a bad rollout of a new runner version by GitHub: https://github.com/actions/runner/issues/3381. We're monitoring the situation.

  • Investigating
    UTC
    Investigating
    We are currently investigating this incident.
GitHub is reporting delays in webhooks which could cause delays in job pickup times
ResolvedDegraded performance1 hour 19 minutes
  • Resolved
    UTC
    Resolved
    This incident has been resolved.
  • Monitoring
    UTC
    Monitoring

    GitHub has reported an incident declaring delays across GitHub services. We are monitoring the situation - https://www.githubstatus.com/incidents/n2969465tx0b

Jun 2024

GitHub reporting degraded performance of Actions
Resolved3 hours
  • Resolved
    UTC
    Resolved
    This incident has been resolved.
  • Monitoring
    UTC
    Monitoring

    We are actively monitoring an incident opened by GitHub about a degraded performance of Actions - https://www.githubstatus.com/incidents/9vwllhs2w1kj

Possible GitHub Actions outage
Resolved1 hour 42 minutes
  • Resolved
    UTC
    Resolved
    This incident has been resolved.
  • Monitoring
    UTC
    Monitoring
    We implemented a fix and are currently monitoring the result.
  • Identified
    UTC
    Identified

    GitHub is reporting degraded service / outage for Actions: https://www.githubstatus.com/incidents/lfrlwdg67fn8

May 2024

Longer queue times for GitHub jobs
Resolved24 minutes
  • Resolved
    UTC
    Resolved
    This incident has been resolved.
  • Identified
    UTC
    Identified

    We've identified the cause of the degradation being some stuck jobs not freeing up CPUs, we're working on a patch to bring queue times back to normal.

  • Investigating
    UTC
    Investigating

    Users may be noticing longer queue times for GitHub jobs, we are investigating the cause of this right now.

Increased GitHub job queue times
ResolvedPartial outage6 hours 29 minutes
  • Resolved
    UTC
    Resolved
    This incident has been resolved.
  • Monitoring
    UTC
    Monitoring

    GitHub is reporting recovery of the GitHub Actions service, we are also observing most jobs being successfully picked up by our Blacksmith runners.

  • Update
    UTC
    Update

    GitHub has opened an official incident to monitor this outage - https://www.githubstatus.com/incidents/nj47vccwm2zj

  • Identified
    UTC
    Identified

    This appears to be a GitHub outage, neither GitHub hosted runners nor Blacksmith hosted runners are being handed jobs. We are continuing to monitor the situation.

  • Investigating
    UTC
    Investigating

    We are noticing an increase in job queue times because of a delay in GitHub handing jobs to our runners

GitHub not assigning jobs to runners
ResolvedPartial outage1 hour 9 minutes
  • Resolved
    UTC
    Resolved
    This incident has been resolved.
  • Identified
    UTC
    Identified

    The issue appears to be on GitHub’s end, they are not assigning jobs to any hosted runners. We will continue to monitor the situation and relay updates.

  • Investigating
    UTC
    Investigating

    We are currently investigating this incident. We are observing GitHub not assigning jobs to hosted runners.

GitHub Action Runners outage
ResolvedMajor outage2 minutes
  • Resolved
    UTC
    Resolved

    GitHub Action Runners is now operational! This update was created by an automated monitoring service.

  • Investigating
    UTC
    Investigating

    GitHub Action Runners cannot be accessed at the moment. This incident was created by an automated monitoring service.

API outage
ResolvedMajor outage2 minutes
  • Resolved
    UTC
    Resolved

    API is now operational! This update was created by an automated monitoring service.

  • Investigating
    UTC
    Investigating

    API cannot be accessed at the moment. This incident was created by an automated monitoring service.

Previous

May 2024 to Jul 2024

Next