ci(queue janitor): never cancel an open pull request's run at its head

Merged
TE
10d ago
+429-139

sr2's order: "A live run on an open PR, at its current head, was cancelled: run 40398277457 (cmux-next, #25446 head 8be738c84d8), attempt 2 cancelled at 22:18:32Z by github-actions[bot]. It held the only Release compile for that head ... fix it so nothing automatic cancels an open PR's run at its head, and add the case to the CI test."

What changed

The janitor (sweep 40410764677, category f, Blacksmith leg path) cancelled #25446's attempt 2. Its re-run had to come from a later sweep, through the macos-pool-load snapshot. The ci-queue-janitor concurrency group keeps evicting pending sweeps, so the run stayed cancelled until a person re-ran it (attempt 3).

  • open_pr_at_head() is now the one predicate every cancelling path checks: categories a-f in build_plan(), classify_lost() (e) and classify_unclaimed() (f) for any event at an open PR's head, and the orphan pass (build_orphan_plan()). Those runs are listed as skipped, with OPEN_HEAD_RULE as the reason ("never cancelled: ... the controller's fall-through moves its compile instead").
  • Unchanged: stale runs (merged, closed or superseded) are still cancelled. Push and dispatch runs at a live branch head with no open PR at that head still get cancelled and re-run once. Pending re-runs left by earlier sweeps are still carried out, since the re-run is what restores the run.
  • So the guard can see every open PR at a head, experiment pushes (category a) and every unprotected orphan's branch (any event) now get a PR lookup in the same batched GraphQL query. Each branch also gets an o<i> alias, pullRequests(states: [OPEN], first: 20), merged into the five newest, so an open PR older than the branch's five newest PRs is still seen.
  • Module docstring updated.

The owned-pool rescue (owned_pool_rescue.py, which cancelled and re-ran #25446's attempt 1) gets the same guard in a separate PR, to keep each PR's test churn small.

Trade-off: lost jobs that now wait

Category e (GitHub never assigning a queued job a runner while newer jobs on the same label start) and category f (an owned or Blacksmith leg unclaimed past its bound) no longer move a run at an open PR's head. That job now waits for the controller's fall-through, GitHub's own queue timeout, or a person. PR #23694's cmux-next swift test on glaeda-gui-std-xcode-26.6 is that case, and its test now expects a skip. Categories c and d only ever judged open-head runs, so in practice they no longer cancel anything.

Risk

Medium. This only removes cancels; nothing new is cancelled. The cost is that a lost job at an open PR's head waits instead of being moved. Merge it once the controller fall-through (cmuxterm-hq, helper "isolated-fallthrough") is deployed.

Tests

python3 -m unittest tests/test_ci_queue_janitor.py passes (220 tests), and so do test_ci_owned_pool_rescue, test_ci_pr_runner_pool, test_ci_starved_labels, test_ci_owned_warm_state and test_ci_health_report.

  • New: test_pr_25446_attempt_2_on_blacksmith_is_never_cancelled. A full main() sweep of a cmux-next same-repo PR run at attempt 2, started by github-actions[bot], with the fast tier queued 50 minutes on blacksmith-6vcpu-macos-26, an App token, and the PR open at that head. The test asserts no POST.
  • New: OpenHeadNeverCancelledTests. One world with a run for every category (a-f) plus the orphan pass, each at an open PR's head on a backed-up pool. It checks the plan, the orphan plan and a full sweep, and asserts no cancel is posted. With the predicate stubbed to None, all of them fail.
  • Changed: tests that expected a cancel at an open head now expect the skip, or now use a live push head with no open PR.

Agent session: agent session

🤖 Generated with Claude Code

Summary by CodeRabbit

  • Bug Fixes
    • Workflow runs associated with the head of an open pull request are no longer automatically cancelled, including orphaned runs and runs from supported event types.
    • When open pull request results are incomplete, cancellation decisions now err on the side of preserving runs.
    • Stale runs on branches without an open pull request can still be cancelled when they meet the existing criteria.
  • Documentation
    • Updated guidance to cover the revised run-handling rules.