ci: rescue hosted jobs cancelled in queue

Merged
TE
11d ago
+93-0

Problem

GitHub-hosted required jobs can be cancelled after waiting about 15 minutes without receiving a runner. Runs 37366561812 and 37366691578 ended this way with zero runner time and no steps, leaving CI fast guards and CLA policy guard red on #17432.

Fix

Add a trusted workflow_run rescue that runs on the Blacksmith Linux pool, recognizes only the no-runner queue-timeout shape after a 14-minute minimum, and reruns the original hosted workflow. It allows at most two retries, ignores runs that had a runner or executed steps, and never cancels an active rescue. The source workflow paths are bound so renames cannot silently disable the watcher.

The rescue only uses the Actions API and never checks out or executes pull-request code. Existing write-then-report workflows retain their non-cancelling concurrency.

Validation

  • actionlint .github/workflows/ci-hosted-queue-rescue.yml
  • python3 tests/test_ci_workflow_run_sources.py
  • python3 tests/test_ci_required_checks_are_bounded.py
  • python3 tests/test_ci_workflow_path_filter_parity.py
  • bash tests/test_ci_self_hosted_guard.sh
  • Shell fixture covering a 905-second no-runner cancellation and rerun

View with [code]smith Autofix with [code]smith
Need help on this PR? Tag @codesmith-bot with what you need. Autofix is disabled.


Summary by cubic

Adds a workflow_run rescue that reruns GitHub-hosted required jobs the Actions service cancels after ~15 minutes in queue without assigning a runner.

  • Detects only the no-runner queue-timeout shape: 14+ minute age with cancelled jobs that have no runner and no steps.
  • Allows at most two retries, ignores runs that had a runner or executed steps, and never cancels an active rescue.
  • Watches CLA policy guard, CLA Assistant, and CI fast guards; source workflow paths are bound so renames fail loudly.
  • Uses only the Actions API and never checks out or executes pull-request code.
  • Forks run the rescue on a standard ubuntu-24.04 runner instead of the internal Blacksmith pool.

Summary by CodeRabbit

  • Chores
    • Added automated recovery for eligible failed or cancelled pull request checks that appear stalled for at least 14 minutes.
    • Recovery is limited to runs with no active runner or completed steps, and to fewer than three attempts. Other failed or cancelled runs are left unchanged.