blacksmith - Elevated failures downloading GitHub release assets – Incident details

Elevated failures downloading GitHub release assets

ResolvedPartial outage6 hours 38 minutes

Affected

Github → Actions

Operational from 6:41 PM to 1:20 AM

  • ResolvedUTC
    This incident has been resolved.
  • MonitoringUTC

    GitHub has implemented a fix and we are beginning to see recovery. We will continue to monitor the issue.

    https://www.githubstatus.com/incidents/lsvy8xsf0gxv

  • UpdateUTC

    GitHub has confirmed an incident affecting release asset downloads, consistent with our findings (githubstatus.com/incidents/lsvy8xsf0gxv). Workflows downloading GitHub release assets remain affected across all regions while GitHub works toward mitigation, and retrying failed jobs remains the most effective workaround. 

  • UpdateUTC

    Workflows downloading GitHub release assets remain affected across all regions because of an upstream issue at GitHub, and retrying failed jobs remains the most effective workaround.

  • UpdateUTC

    Workflows downloading GitHub release assets remain affected across all regions, and retrying failed jobs remains the most effective workaround. The failures remain isolated to a subset of GitHub's edge servers upstream of our infrastructure, and we are continuing to monitor for recovery.

  • UpdateUTC

    We have isolated the cause to a subset of GitHub's edge servers and verified the failures occur from networks outside Blacksmith as well. Workflows downloading GitHub release assets remain affected across all regions, and retrying failed jobs is the most effective workaround until the upstream issue is resolved. We will provide another update within the next 30 minutes.

  • UpdateUTC

    We are still experiencing an elevated rate of socket hangup failures when workflows download GitHub release assets, across all regions. The failures remain isolated to GitHub upstream, and we are continuing to monitor for recovery; other GitHub interactions, including repository checkout, remain unaffected. We will provide another update within the next 30 minutes.

  • IdentifiedUTC

    We are still experiencing an elevated rate of socket hangup failures when workflows download GitHub release assets, across all regions. The failures have been isolated to GitHub upstream, and we are monitoring closely while the issue persists. We have observed multiple public reports that GitHub is experiencing a partial outage.

  • InvestigatingUTC

    We are investigating reports of socket hangups when attempting to fetch GitHub release assets. There are not any current upstream incidents, but we are continuing to investigate.