ci: rescue hosted jobs cancelled in queue
Problem
GitHub-hosted required jobs can be cancelled after waiting about 15 minutes without receiving a runner. Runs 37366561812 and 37366691578 ended this way with zero runner time and no steps, leaving CI fast guards and CLA policy guard red on #17432.
Fix
Add a trusted workflow_run rescue that runs on the Blacksmith Linux pool, recognizes only the no-runner queue-timeout shape after a 14-minute minimum, and reruns the original hosted workflow. It allows at most two retries, ignores runs that had a runner or executed steps, and never cancels an active rescue. The source workflow paths are bound so renames cannot silently disable the watcher.
The rescue only uses the Actions API and never checks out or executes pull-request code. Existing write-then-report workflows retain their non-cancelling concurrency.
Validation
actionlint .github/workflows/ci-hosted-queue-rescue.ymlpython3 tests/test_ci_workflow_run_sources.pypython3 tests/test_ci_required_checks_are_bounded.pypython3 tests/test_ci_workflow_path_filter_parity.pybash tests/test_ci_self_hosted_guard.sh- Shell fixture covering a 905-second no-runner cancellation and rerun
Need help on this PR? Tag @codesmith-bot with what you need. Autofix is disabled.
Summary by cubic
Adds a workflow_run rescue that reruns GitHub-hosted required jobs the Actions service cancels after ~15 minutes in queue without assigning a runner.
- Detects only the no-runner queue-timeout shape: 14+ minute age with cancelled jobs that have no runner and no steps.
- Allows at most two retries, ignores runs that had a runner or executed steps, and never cancels an active rescue.
- Watches
CLA policy guard,CLA Assistant, andCI fast guards; source workflow paths are bound so renames fail loudly. - Uses only the Actions API and never checks out or executes pull-request code.
- Forks run the rescue on a standard
ubuntu-24.04runner instead of the internal Blacksmith pool.
Summary by CodeRabbit
- Chores
- Added automated recovery for eligible failed or cancelled pull request checks that appear stalled for at least 14 minutes.
- Recovery is limited to runs with no active runner or completed steps, and to fewer than three attempts. Other failed or cancelled runs are left unchanged.