ci(owned-pool rescue): open-head failover for runs with finished or red siblings, judged by real label starts

Merged
TE
7d ago
+1145-106

Leo, 2026-10-28 (rulebook, CI and GitHub): "Failover is stall-aware: a job queued more than about 3 minutes on a label that has started nothing is cancelled and re-dispatched onto a live member of the set; this failover never cancels a started nightly, dev or promote build." Target: zero queue.

This PR carries dep-fleet-2's commit 4cbd22a0 (branch lane/fasttier-open-head-replace, with its authorship) plus the settled-sibling failover on top. Their snapshot-token fix is left out because it lands on its own in #28163.

What was wrong

From 2026-10-29 to 10-30 the open-head failover (#27521) moved no run. Three cases:

  • 40997814347 (#27853): cmux-next Release compile (Xcode 26) was queued on glaeda-std-xcode-26.6 from 03:35Z, over 4.5 h. 28 of the run's 29 jobs had finished, with its swift test and daemon tests red. At 08:04Z ci-dash showed the label with 40 queued, the oldest 110 min, and 0 running.
  • 40993742146 (#27797) and 41021261503 (#27756): their fast tiers waited hours on glaeda-side-std and glaeda-std behind a red checks.

Five reasons:

  1. Finished jobs blocked the failover. It required that nothing but placement jobs had finished. A cmux-next PR run's Linux jobs always finish before a Mac job has waited 3 minutes.
  2. Red siblings blocked the rescue. With a red sibling, assess(leave_decided=True) returned "watch", so the failover was never consulted. The 05:10Z sweeper logged 171 such looks for 40993742146.
  3. "Started nothing" meant running == 0. That count includes root rollups and jobs started hours ago. glaeda-side-std showed running=1 from a ~02:00Z job while 17 jobs sat queued.
  4. The re-run went back to the dead label. A job re-run with its old placement sends cmux-next bot attempt 2 back to CI_SIDE_LANE_RUNNER.
  5. Repeating red Mac jobs blocked the placement re-run. Run 40997814347's red swift test and daemon tests sit below the placement, so re-running the placement would repeat them, and the failover kept the open-head rule.

What changed (scripts/ci/owned_pool_rescue.py, queue_janitor.py)

From 4cbd22a0 (dep-fleet-2):

  • Placement re-run. placement_rerun() / failover_loses(): a cmux-next PR run's attempt 1 fails over by re-running cmux-next macOS placement (POST /actions/jobs/{id}/rerun). The placement and its dependents re-place afresh, and every other finished job keeps its result, a red checks included. A queued job counts as coming back only when every workflow job its name could be needs the placement (PlacementRerun.brings_back()); an unmatched or ambiguous name keeps the run where it is. If GitHub refuses the placement re-run, the rescue reports the refusal and stops; it never widens to a re-run of every job.
  • The watch offers the failover. watch() offers it every FAILOVER_LOOK_SECONDS (120 s) to a PR run with a Mac job queued at least 180 s, whatever kept the queue rescue off: a red sibling, refusals-only, or a hold. follow() treats that as a rescue.
  • Dead labels are judged by real starts. The janitor snapshot gains label_started, the newest start of a runner-taken job on each exact macOS label. label_quiet() counts a label dead when it has started nothing since the job began waiting, judged from a snapshot made at least 180 s after that.

Added here:

  • Settled-sibling failover (open_head_failover() → Failover(settled=True)). It applies when three things hold:

    • the placement or full re-run would repeat a finished job;
    • no job of the run is running or held by a runner (job_running());
    • the stalled job is the attempt's one unfinished job.

    The cancel then stops only that job, which never ran. Every finished job keeps its result, red ones included. Only that job and its dependents run again (rerun_job). If a second job is still queued (a Linux job, or another Mac job), the open head keeps the run, because no single re-run is shown to bring back every job the cancel stops. A later look moves the run once only the stalled job is left.

    If GitHub refuses the single-job re-run, the rescue reports the refusal and stops. It does not fall back to re-running the failed jobs, which would repeat the finished failures the failover keeps.

    settled_guard() reads the run, then the jobs, right before the cancel and before every re-send. Any of these drops the cancel: a job that started or was assigned, another job that queued meanwhile, the stalled job finishing or leaving the listing, a new attempt, or a finished run. The jobs read comes after the run read, so a job that starts during a slow run read still drops it. The receipt names the waiting jobs cancelled and the finished jobs kept. It is given only once the cancel went out, for both failovers, so a dropped cancel leaves no receipt. The CANCEL_SENT mark carries the job the re-run starts from (of job N): the waiting job, or a placement failover's placement job. A successor of a rescue that stopped between its cancel and its re-run therefore re-runs that job and its dependents, never every job or every failed job.

  • Both failovers everywhere. watch()'s periodic offer and the refusals-only check consider both failovers.

  • Exclusions are unchanged (never_fails_over() etc.): only PR runs fail over, never nightly, DEV, promote, feed, dispatch or push runs.

The settled path in cmux-next. A bot attempt 2 or 3 of a job re-run takes the side label, and attempt 4 takes Blacksmith (next_attempt()). If the side label is dead too, each later attempt fails over again after 3 minutes. The placement re-run, which re-places afresh, is used whenever it loses nothing.

Review findings (CodeRabbit)

Fixed (CodeRabbit, then an Astra review that blocked an earlier head), each with a test that is red without the fix:

  • The guard re-reads the jobs after the run read.
  • A refused single-job re-run returns the refusal, with no fallback; so does a refused placement re-run.
  • A queued job with an unmatched or ambiguous name is never cancelled by a placement failover.
  • The resume path keeps the waiting job's id, and a placement failover's mark names the placement job instead of every job.
  • A settled cancel never leaves a queued job behind: with a second job queued, the open head keeps the run.
  • The receipt is emitted only after the cancel.

Skipped: the settled re-run keeps its old placement (Major, heavy lift). A job re-run keeps the finished placement's outputs, so a cmux-next bot attempt 2 or 3 can queue on a dead side label again. Fixing that means re-placing without repeating the placement's red dependents, which GitHub's re-run API cannot do. The cost is bounded: the next attempt fails over again after 3 minutes, and attempt 4 goes to Blacksmith (next_attempt()). Where the placement re-run loses nothing, it is used instead, and it re-places afresh.

Class sweep

Two classes came out of review. Every site of each in scripts/ci/owned_pool_rescue.py at this head, carried code included:

(a) A cancel, or a decision to cancel, made from a job listing read before another API read that can be slow

  • open_head_guard() (the placement and full-re-run failover's last check before the cancel). It read the jobs, then the run. Fixed: it now reads the run, then the jobs. A job that queues during a slow run read now drops the cancel (new test).
  • settled_guard(): reads the run, then the jobs. Fixed in an earlier round, with a test.
  • nightly_cancel()'s nightly check (main's code) reads the jobs, then the run, on purpose. If the run read waited (rate limit, or too slow), it reads both again, up to NIGHTLY_GUARD_TRIES, and drops the cancel when they stay unconfirmed. Safe; unchanged.
  • open_head_failover() decides from its look's listing, then reads the workflow file (failover_plan()). It never cancels. Every failover cancel goes through one of the two guards above, which re-read right before the cancel and before every re-send.
  • failover_receipt() reads the jobs before the guard. It is text only, emitted after the cancel went out, and decides nothing.
  • rescue()'s started_dev_compile() check (main's code) reads the jobs before started_nightly_work() and the cancel. On the failover paths, the guards re-read and drop the cancel if any job started (a DEV compile included). The plain queue rescue is main's and is unchanged here.
  • hold_for_destination()'s turned_away read, follow()'s refusal_jobs read and every watch() read lead to an outcome, never a cancel. A cancel only comes later, through rescue() and its guard. The refusal check in follow() can only decline.

(b) A watch() branch that sleeps, continues or returns before the failover offer

  • watch()'s refusal wait (look.action == "refused", run unfinished) slept and continued before the offer. Fixed: the offer now comes right after assess(), before this wait. It is made when the run also failed on its own, because follow() would then decline the refusal path (new test). A refusal with no failure of its own keeps the refusal path, whose re-run of failed jobs brings the refused job back.
  • watch()'s return look.action for "rescue" and "refused" came before the offer. A hold for a Blacksmith slot (hold_for_destination(), which sleeps STALL_RECHECK_SECONDS and watches again) so never reached it. Fixed: the offer comes first; a "rescue" look is offered too.
  • watch()'s nightly refusal return ("watch limit reached with a started nightly still running") comes after the offer. A nightly never fails over (never_fails_over()).
  • watch()'s branches before the pool choice (not on_persistent: picker, late placement, re-run of failed jobs) stop or poll before any Mac job is placed, so there is no queued Mac job to offer for yet. The finished-run branches (the run finished, cancelled, resume) have nothing left to move.
  • follow()'s refusals-only re-watch and its NIGHTLY_BUSY re-watch call watch(), so they reach the offer. settle()'s signing-stall loop (sleep(POLL_SECONDS)) only covers nightly signing jobs, which never fail over.

Tests

python3 -m unittest tests.test_ci_owned_pool_rescue: 401 OK, rebased on main with #26332. test_ci_queue_janitor and test_ci_starved_labels are also OK.

Red at 4cbd22a0 alone, green here:

  • OpenHeadSettledFailover: run 40997814347 re-runs only its compile and keeps 28 finished jobs; a second queued job, Mac or Linux, keeps the open head, and one that queues before the cancel drops it, as does the stalled job finishing or leaving the listing; a placement failover stopped after its cancel resumes only the placement; a refused job re-run is reported and never widened; a job that starts during the guard's run read drops the cancel; a resumed rescue re-runs the job its mark names; follow() moves a dead label beside a refused sibling and a failure of its own, and behind red siblings with Blacksmith stalled, refusals-only or not. Also covered: a running, assigned or in-setup job keeps the open head, a busy label keeps it, a job that starts before the cancel drops it and leaves no receipt, and nightly, DEV, dispatch and push targets never fail over.
  • Six existing subtests that kept the open head where the re-run would repeat a finished job now expect the settled path. Running or assigned jobs still keep the open head.

Placement tests added here: a job that queues while open_head_guard() reads the run drops the cancel; a refused placement re-run is reported, never widened; a stalled Mac job beside a queued job with an unmatched or ambiguous name is not cancelled.

4cbd22a0's own tests (placement subtree, label_quiet, the red-sibling watch offer, the janitor's label_started) carry over unchanged.

No new timer or loop. The watch's offer reuses its existing poll, and the failover's second look is the existing bounded sleep.

Agent session: agent session

🤖 Generated with Claude Code