ci(owned-pool rescue): never cancel a run whose pull request is open at its head

Merged
TE
11d ago
+148-43

sr2's order: "A live run on an open PR, at its current head, was cancelled: run 40398277457 (cmux-next, #25446 head 8be738c84d8) ... fix it so nothing automatic cancels an open PR's run at its head, and add the case to the CI test." The coordinator extended the order to the owned-pool rescue. The janitor half is #25746.

What changed

scripts/ci/owned_pool_rescue.py, rescue(): #25446's attempt 1 was cancelled here (cancel, then an in-process re-run) at 21:23Z. Attempt 2 was then cancelled by the janitor and left cancelled.

  • Once pull_moved() has confirmed the pull request is open at the run's head, a run that is still in flight is no longer cancelled. The rescue returns not rescued: PR #N is open at the run's head; <OPEN_HEAD_RULE>, and the watch logs it. This covers every target kind: CI, side lane, cmux-next, test-ios, and any attempt.
  • Unchanged: a run that already finished still has its failed jobs re-run, with no cancel (a refused job, for example). Superseded and closed pull requests are handled as before. Runs with no pull request (main's full-suite dispatch, nightly, E2E and side-lane pushes and dispatches) keep the cancel and re-run.
  • The module docstring says this.

Trade-off

A job stuck queued on an owned label (or refused while siblings still run) in a pull request run at its open head now waits. It is no longer moved to Blacksmith by cancel and re-run. The controller's fall-through moves the compile instead. A refusal's failed jobs are still re-run once the run finishes.

Risk

Medium. This only removes cancels. Merge it once the controller fall-through (cmuxterm-hq, helper "isolated-fallthrough") is deployed.

Tests

python3 -m unittest tests/test_ci_owned_pool_rescue.py passes (191 tests), and so do test_ci_queue_janitor, test_ci_pr_runner_pool, test_ci_late_placement and test_ci_owned_pool_sources.

  • New: OpenHeadNeverCancelled. It covers the #25446 cmux-next run at its open head with a stuck fast tier, and every target kind with failed_only on and off (no cancel or re-run, only pull and run reads). A finished run still gets its failed jobs re-run, and superseded or closed runs are unchanged. With the guard disabled, all of them fail.
  • Changed: the cancel-mechanics tests (settle wait, force-cancel, rate-limit window, deadline margin, someone else re-ran, push during the cancel, queue budgets) now drive main's dispatch, which still cancels. Tests that watched a pull request run to a cancel now assert that it was left alone at its open head (assert_left_at_open_head). Side-lane cancels use a dispatch with no pull request.

Agent session: agent session

🤖 Generated with Claude Code

View with [code]smith Autofix with [code]smith
Need help on this PR? Tag @codesmith-bot with what you need.

Summary by CodeRabbit

  • Bug Fixes
    • CI runs for pull requests that remain open at the run’s current commit are no longer cancelled while in progress.
    • Stuck jobs remain queued for controller handling. After a run finishes, failed jobs from refused pull requests can still be retried. Superseded or closed pull requests retain their existing handling.