ci(owned-pool rescue): stall-aware failover at an open head only when nothing has started

Merged
TE
8d ago
+501-12

Why

Leo, 2026-10-28 (rulebook, CI routing): "Failover is stall-aware: a job queued more than about 3 minutes on a label that has started nothing is cancelled and re-dispatched onto a live member of the set." Since #25753 the owned-pool rescue never cancels a pull request's run at its open head (OPEN_HEAD_RULE), so for PR runs that failover never happened. Run 40887257098 (#26999, attempt 2) had its daemon test queued on glaeda-session-std-xcode-26.6 from 15:20Z to 18:41Z. No runner carrying the label was online, and nothing moved it.

Two things kept it there:

  1. The rescue returned OPEN_HEAD_RULE for every in-flight PR run.
  2. The queue janitor's macos-pool-load snapshot never counted feat-cmux-next's lane labels (session, fast, dev, nightly). No snapshot reader could tell that the label ran nothing. That includes cmux-next's placement, which reads queue depth from the snapshot.

Policy

dep-fleet's decision under Leo's 2026-10-28 failover order ("Failover is stall-aware: a job queued more than about 3 minutes on a label that has started nothing is cancelled and re-dispatched onto a live member of the set; this failover never cancels a started nightly, dev or promote build"), verbatim:

Policy call on cmux#27521: (A). The intent of rule (a) was "nothing of value is lost by the cancel", and placement jobs lose nothing: a re-run repeats about a minute of Linux placement, and that's all. Encode it exactly like this:

  • A fixed, named list of placement or preflight jobs (path routing, macOS placement, head preflight, ci.yml changes), declared in one place with a test that fails if the list gains a job that builds, tests or signs.
  • Those jobs may be finished, but only with conclusion success. A failed or cancelled placement job means no failover.
  • Every other job in the run must be queued or waiting, with no runner assigned (runner_id/runner_name empty). If any non-listed job has started, no failover.
  • Never for nightly, dev or promote workflows (keep that exclusion explicit).
  • Record what was cancelled and why in the run's re-dispatch receipt.

The other conditions from the original order: (b) the stall-aware condition holds (a job queued more than about 3 minutes on a label that has started nothing), and (c) the rescue immediately re-runs the same run at the same head, so the attempt is re-placed with a health check. Every other open-head case keeps OPEN_HEAD_RULE.

How the code maps to it: PLACEMENT_JOBS (the four, one place; test_the_placement_list_holds_only_placement_jobs); job_started() (listed jobs only with conclusion success; every other job queued, waiting, pending or requested with neither runner_name nor runner_id; a job GitHub skipped, which never had a runner, counts as never started); NEVER_FAILOVER_PATHS plus any path naming nightly (never_fails_over()), next to the existing nightly, main, dispatch and push exclusions; failover_receipt() (a log line and a ::notice annotation naming the queued jobs cancelled, the placement jobs repeated, and why).

What changes

  • owned_pool_rescue.unstarted_failover(): at an open head, the rescue may cancel and immediately re-run every job at the same head only when all of these hold:

    • no macOS job of the run has been taken by a runner (this attempt or a carried earlier one), and no job is running;
    • a job has waited at least UNSTARTED_STALL_SECONDS (180 s) for a runner on a label that the janitor's fresh snapshot lists with nothing running (a label it does not list proves nothing);
    • the run is a pull request's and is not a nightly, DEV feed or dispatch build.

    A job that waited less is looked at once more when it reaches 180 s. The cancel goes through the existing guarded cancel. A guard re-reads the jobs right before the cancel (and every re-send) and drops it if any job was taken meanwhile. The re-run is a full one, so the placement runs again and checks the routes. The PR head is re-checked before the re-run, as before. Every other open-head case keeps OPEN_HEAD_RULE.

  • A refusals-only watch (a cmux-next run whose placement found no free Blacksmith slot) used to ignore queue alarms. It now takes the same failover when the label runs nothing, since a dead label is not a queue to wait in.

  • pr_runner_pool.LANE_LABEL is the one definition of the lane labels. The rescue imports it, and queue_janitor.pool_load_snapshot() now counts queued and running jobs on lane labels. Pickers that sum owned families (simple_pool_picker.owned_family) don't match lane labels, so their numbers don't change.

Interpretation (superseded by the Policy above)

"No job has started" is read as no running job and no macOS job taken by a runner. Linux gate and placement jobs finish before the Mac jobs exist, so they can't be "unstarted"; a full re-run repeats them (a few minutes of Linux). A re-run of failed jobs that carries a finished Mac job (like 40887257098's attempt 2, which carried its swift test) does not qualify, because a full re-run would discard that result. That case belongs to the placement fix (a person's rerun-failed reusing attempt 1's placement), which comes as a separate PR on feat-cmux-next.

Risk

Medium. This adds a cancel path for PR runs, but only when no Mac job has started. The guard drops the cancel if one starts. A nightly, DEV or dispatch run never takes the path. Extra cost is one snapshot read and one or two job listings per stuck open-head PR run. No workflow, permission or trust change.

Tests

  • New OpenHeadUnstartedFailover (9 tests): run 40887257098's shape is moved (cancel, then a full rerun). Each of these keeps OPEN_HEAD_RULE: a label that runs a job, a label the snapshot doesn't list, a running Mac job, a carried finished Mac job, a running Linux job, a job that starts before the cancel, and nightly, DEV-daily or iOS dispatch targets. A short wait is re-looked at 180 s. A refusals-only watch fails over a dead label but not a live one. The janitor snapshot counts a session-label job.
  • python3 -m unittest tests.test_ci_owned_pool_rescue: 345 passed. tests.test_ci_queue_janitor: 243 passed. test_ci_pr_runner_pool.py, test_ci_simple_pool_picker.py and test_ci_starved_labels.py pass.

Changelog

none

šŸ¤– Generated with Claude Code

Agent session: agent session

Summary by CodeRabbit

  • Bug Fixes
    • Open pull-request checks can now recover from a stalled, unstarted macOS job by cancelling and rerunning the full check suite when the runner pool is clear.
    • The recovery is skipped if a job has started, the run has changed, or job status information is incomplete. Nightly, development, and dispatch runs are excluded.
    • Queue status now includes queued and running jobs across supported runner lanes, improving visibility into wait times and pool load.