blacksmith - Delayed job starts on macOS runners – Incident details

The queue for macOS jobs is subsiding as restored capacity continues to come online with our upstream provider, and new 6vcpu and 12vcpu macOS jobs are picking up normally for most customers. A small number of 12vcpu macOS jobs queued earlier today are still being worked through and may take longer to start, while Linux and Windows runners are unaffected. We are continuing to monitor and will resolve this incident once the remaining backlog has cleared.

New macOS jobs are starting within a few minutes for most customers, though some 6vcpu macOS jobs may wait up to 10 minutes during busy periods while we finish restoring capacity with our upstream provider. A backlog of 12vcpu macOS jobs queued earlier today is still being worked through, and those jobs may take longer to start, while Linux and Windows runners are unaffected.

6vcpu macOS jobs are now starting within a few minutes, and the backlog of 12vcpu macOS jobs is draining as we continue to add capacity with our upstream provider. Customers with 12vcpu macOS jobs already queued may still see longer waits for them to start, while new jobs are picking up faster, and Linux and Windows runners are unaffected

We have brought additional macOS capacity online and queue times have started to improve. Customers should currently expect 6vcpu macOS jobs to wait a few minutes to start and 12vcpu macOS jobs considerably longer, while Linux and Windows runners are unaffected.

We are continuing to bring macOS hosts back into service and are working with our upstream provider to release the remaining hosts. Customers should currently expect 6vcpu macOS jobs to wait around 10 minutes to start and 12vcpu macOS jobs considerably longer. Linux and Windows runners are unaffected.

We are continuing to bring macOS hosts back into service in batches and are working with our upstream provider to release the remaining hosts. Customers should currently expect 6vcpu macOS jobs to wait around 10 minutes to start and 12vcpu macOS jobs considerably longer, while Linux and Windows runners remain unaffected.

We have returned an additional group of macOS hosts to service and expect a further large batch to come online within the next 30 minutes, at which point waiting macOS jobs should start being picked up more quickly. Customers should currently expect 6vcpu macOS jobs to wait around 5 minutes to start, while 12vcpu macOS jobs are waiting considerably longer, and Linux and Windows runners are unaffected.

We are continuing to bring additional macOS hosts back online with our upstream provider, and the capacity restored earlier remains in service. Customers may still see longer than normal queue times for macOS jobs, with 12vcpu macOS jobs most affected, while Linux and Windows runners are unaffected.

We have restored a portion of the affected macOS capacity and are continuing to work with our upstream provider to bring the remaining hosts back online. Customers should still expect longer than normal queue times for macOS jobs, with 12vcpu macOS jobs most affected, while Linux and Windows runners are unaffected.

We are continuing to work with our upstream provider to bring the affected macOS hosts back online. Customers should still expect longer than normal queue times for macOS jobs. We will provide another update shortly.

We are continuing to bring additional macOS hosts online with our upstream provider however available macOS capacity has not yet recovered. Customers can still expect longer than normal queue times for macOS jobs, with 12vcpu macOS jobs most affected. We will provide another update shortly