#26614 ci(nightly-next): queue on the busy owned pool, not a stalled Blacksmith label

Leo: "it is unacceptable to have nightlies >30min out of date." Routing order: Linux, then owned Macs, then Blacksmith macOS last ("IT WAS ALWAYS FUCKING LAST RESORT").

What changed

scripts/ci/nightly_build_runner.py picks the macOS runner for nightly-next's build-nightly-app and, in verify mode, for the verify jobs. When no owned runner was idle, it always fell back to Blacksmith, even when every Blacksmith macOS label was stalled. Now, when no owned runner is idle and a fresh pool snapshot sees every Blacksmith macOS 26 candidate stalled, it queues on the first watched owned label that an online runner carries.

That is safe because the owned-pool rescue already holds a job queued on its owned pool while the Blacksmith label its re-run would take is stalled, and moves it once that label moves (owned_pool_rescue.hold_for_destination()). So the job is never stranded on a label with no runner:

• offline-only pools, unread runners (no routing token) and a missing snapshot all still go to Blacksmith;
• if any Blacksmith label still moves, it takes the job as before.

Before / after

Before: run 40622241620 (nightly-next @ 0c833639) had pick-nightly-build-runner print owned: none (no idle runner carries glaeda-nightly-std-xcode-26.6 ...) and every Blacksmith macOS label is stalled. Its build-nightly-app then waited on blacksmith-6vcpu-macos-26 from 02:40:33 to 02:55:33Z (15 min, behind the classic main nightly queue). Over the same window, the owned nightly slots picked up the next tips' builds at 02:53:18 (cmux13s) and 02:57:58 (cmux12s).
After: the same inputs give owned: glaeda-nightly-std-xcode-26.6, so the job queues on the owned slot, which frees within one build. New test: test_every_blacksmith_label_stalled_queues_on_the_busy_owned_pool. The verify-mode test that pinned the old behaviour is updated.

Tests: tests/test_nightly_build_runner.py (33 OK), tests/test_nightly_trusted_routing.py (20 OK), tests/test_nightly_sign_runner.py (17 OK).

Risk: low. Two picker functions change, and only in the case where every Blacksmith label is stalled.

Agent session: agent session

🤖 Generated with Claude Code

---

^{Need help on this PR? Tag @codesmith-bot with what you need.}

---

Summary by cubic

Prevents nightly-next builds from waiting on a stalled Blacksmith macOS label when no owned runner is idle.

Previously, the runner picker always fell back to Blacksmith when no owned runner was idle, even when every Blacksmith macOS label was stalled. Now, when a fresh snapshot shows every Blacksmith label stalled, the build and verify picks queue on the first watched owned label an online runner carries, which frees within one build. If any Blacksmith label still moves, it takes the job as before; offline-only pools, unread runners, and missing snapshots still route to Blacksmith. This is safe because the owned-pool rescue holds the job until the stalled label moves.

• Adds owned_queue_route() and blacksmith_stalled_everywhere() to scripts/ci/nightly_build_runner.py.
• Extends tests/test_nightly_build_runner.py with the stalled-everywhere and no-rescue cases.