blacksmith - GitHub Actions checkout are taking longer than usual – Incident details

eu-west Storage Cluster under maintenance

GitHub Actions checkout are taking longer than usual

Resolved
Degraded performance
Started about 1 year agoLasted 24 hours 53 minutes

Affected

Github → Actions

Operational from 10:10 PM to 11:03 PM

Updates
  • Resolved
    UTC
    Resolved

    This incident has been resolved. We will be providing a postmortem to affected customers by tomorrow.

  • Monitoring
    UTC
    Monitoring
    We implemented a fix and are currently monitoring the result.
  • Identified
    UTC
    Identified
    We are continuing to work on a fix for this incident.
  • Monitoring
    UTC
    Monitoring

    Our upstream ISP identified some issues between them and their network vendor. They have made changes to mitigate these blips while working on a longer term resolution. We will share more details as we receive them. All network operations in Blacksmith runners have returned to normal. We are actively monitoring the situation.

  • Update
    UTC
    Update

    We are being alerted about intermittent degradation in our network stack in one of our regions. This may be sporadically manifesting as slower docker pulls, interactions with certain language mirrors, git checkouts. We are investigating the source of this issue.

  • Investigating
    UTC
    Investigating
    We're hearing this from several customers today that they are seeing elevated `actions/checkout` times. We are noticing this in our local development as well. This is very likely an upstream degradation with the GItHub control plane but we are investigating what we can do to mitigate these.