#28471 ci(owned-pool rescue): the sweeper watches queued side-lane runs, not only failed ones

Order (rulebook, Leo 2026-10-28): "a job queued more than about 3 minutes on a label that has started nothing is cancelled and re-dispatched onto a live member of the set." And: "No job waits on one reserved machine."

What happened

Run 41055587932 (nightly-next-launch-smoke.yml, a pull request run at head c315a540) put its smoke job on glaeda-side-std-xcode-26.6 at 12:28Z on 2026-10-30. That label had 0 of 4 runners online. The job stayed queued for more than 5 hours. Two more pull request runs of the same workflow (41090565782, 41105677856) are stuck on that label the same way. The rescue never moved them because it never looked at them:

• A pull request's pick mints no routing token, so it skips side_lane_placement.py and the smoke takes vars.CI_SIDE_LANE_RUNNER without checking it. That is by design: the rescue is supposed to catch it.
• The rescue that actually runs is main's. workflow_run and schedule triggers register only from the default branch, and the job checks out ref: main. main's ci-owned-pool-rescue.yml gets no workflow_run event for most side lanes (#16076), including the launch smoke. feat-cmux-next's copy of that file lists the side lanes, but nothing ever reads that copy.
• main's sweeper finds side-lane runs only through failed_runs(), which lists failed and cancelled runs. A job waiting on a label that starts nothing never fails its run. Neither sweeper that ran over that window (10:44Z to 14:08Z, and 16:06Z onward) mentions 41055587932 anywhere in its log.

#28374 already lets a pull request run of the launch smoke or the canary use the open-head failover (test_a_pull_request_run_of_the_canary_or_launch_smoke_fails_over_a_dead_label). But nothing ever started a watch on such a run.

Change

• New GitHub.queued_side_runs(). It lists status=queued runs of SIDE_WORKFLOW_PATHS and skips two kinds: cmux-next.yml, whose placement's marker already adopts it, and delivery runs (never_fails_over(), for example a scheduled launch smoke), so nothing here cancels a delivery run that has not failed. Paging works like failed_runs(): only an empty page ends it early.
• sweep() calls it in the same tick and under the same budget gate as the failed-run listing (every SIDE_FAILURE_TICKS ticks). It adopts each run once per attempt, and skips a run its marker or the re-run listing already follows. Once adopted, follow()/watch() decide as before: the open-head failover runs at an open PR head, the queue rescue otherwise.
• Cost: at most SWEEP_LISTING_PAGES extra requests every 5 minutes (usually one), plus a watch per stuck run. No new timer or loop. The listing runs inside the sweeper's existing tick loop, which ends at SWEEP_SECONDS or a code handover.

Not fixed here (outside this code)

• Scheduled (delivery) smokes on main are stuck too: 7 on the side label and 5 on blacksmith-6vcpu-macos-26. The 17:06Z pick logged Blacksmith macOS as stalled: blacksmith-12vcpu-macos-26 0 in flight + 196 queued (oldest 1419 min), blacksmith-6vcpu-macos-26 0 in flight + 92 queued (oldest 687 min). When the cascade sees that, it keeps the job on the side label, even though that label has no runner online. This PR leaves delivery runs alone, by the dep-fleet ruling.
• Even after the failover, a moved smoke's attempt 2 goes to vars.MACOS_RUNNER_26 || blacksmith-6vcpu-macos-26. Until Blacksmith macOS starts jobs again, that is not a live destination.

Tests

The tests were written first and failed on base ([] != [(1, 1, False)]):

• test_a_queued_side_lane_run_is_adopted_without_a_marker_or_a_failure
• test_a_queued_side_lane_run_already_watched_through_its_marker_is_not_adopted_again
• test_queued_side_runs_lists_queued_side_lane_runs_only

python3 -m unittest tests.test_ci_owned_pool_rescue: 408 tests OK. tests.test_ci_owned_pool_sources: OK.

Prior art: #28249 (into feat-cmux-next, on HOLD) adds side-run discovery to feat-cmux-next's owned_pool_rescue.py, which no rescue run executes. This PR makes the discovery part of that change on main, where the rescue runs. #28431 (main) changes other functions.

Risk: low. The PR only adds watches. The cancel and re-run rules a watch applies are unchanged.

Agent session: agent session

🤖 Generated with Claude Code

---

^{Need help on this PR? Tag @codesmith-bot with what you need.}

---

Summary by cubic

Fixes the CI sweeper so queued side-lane runs get watched, instead of only failed ones. A job waiting on a label where no runner starts never fails its run, so before this it was never watched and the open-head failover never moved it (run 41055587932 waited over five hours on an offline label).

• Adds GitHub.queued_side_runs() to list queued runs of SIDE_WORKFLOW_PATHS, paged like failed_runs().
• Calls it in the same tick and budget gate as the failed-run listing, and adopts each run once per attempt.
• Skips cmux-next.yml runs (their marker already adopts them) and delivery runs such as scheduled smokes, so nothing here cancels a delivery run that has not failed.

Summary by CodeRabbit

• New Features
  • Queued side-lane runs are now picked up for monitoring, including runs without a failure or marker.
  • Runs already being monitored are not picked up again, and temporary read failures can be retried.