blacksmith - Elevated failures downloading GitHub release assets – Incident details

Elevated failures downloading GitHub release assets

Resolved
Partial outage
Started 19 days agoLasted about 7 hours

Affected

Github → Actions

Operational from 6:41 PM to 1:20 AM

Updates
  • Resolved
    UTC
    Resolved
    This incident has been resolved.
  • Monitoring
    UTC
    Monitoring

    GitHub has implemented a fix and we are beginning to see recovery. We will continue to monitor the issue.

    https://www.githubstatus.com/incidents/lsvy8xsf0gxv

  • Update
    UTC
    Update

    GitHub has confirmed an incident affecting release asset downloads, consistent with our findings (githubstatus.com/incidents/lsvy8xsf0gxv). Workflows downloading GitHub release assets remain affected across all regions while GitHub works toward mitigation, and retrying failed jobs remains the most effective workaround. 

  • Update
    UTC
    Update

    Workflows downloading GitHub release assets remain affected across all regions because of an upstream issue at GitHub, and retrying failed jobs remains the most effective workaround.

  • Update
    UTC
    Update

    Workflows downloading GitHub release assets remain affected across all regions, and retrying failed jobs remains the most effective workaround. The failures remain isolated to a subset of GitHub's edge servers upstream of our infrastructure, and we are continuing to monitor for recovery.

  • Update
    UTC
    Update

    We have isolated the cause to a subset of GitHub's edge servers and verified the failures occur from networks outside Blacksmith as well. Workflows downloading GitHub release assets remain affected across all regions, and retrying failed jobs is the most effective workaround until the upstream issue is resolved. We will provide another update within the next 30 minutes.

  • Update
    UTC
    Update

    We are still experiencing an elevated rate of socket hangup failures when workflows download GitHub release assets, across all regions. The failures remain isolated to GitHub upstream, and we are continuing to monitor for recovery; other GitHub interactions, including repository checkout, remain unaffected. We will provide another update within the next 30 minutes.

  • Identified
    UTC
    Identified

    We are still experiencing an elevated rate of socket hangup failures when workflows download GitHub release assets, across all regions. The failures have been isolated to GitHub upstream, and we are monitoring closely while the issue persists. We have observed multiple public reports that GitHub is experiencing a partial outage.

  • Investigating
    UTC
    Investigating

    We are investigating reports of socket hangups when attempting to fetch GitHub release assets. There are not any current upstream incidents, but we are continuing to investigate.