PR #27095 - manaflow-ai/cmux

What broke

GitHub fixes a job's runner label when it creates the job. If that label stops serving, the job waits forever, and changing the runner repository variables does not move it. Nightly-next push run 40771197361 (tip 1f0f687b, 2026-10-28) had decide waiting 34 min on blacksmith-4vcpu-ubuntu-2404, with more jobs queued behind it on the same label. Between about 20:00Z and 21:15Z every Blacksmith Linux label stopped starting jobs. The feed watch stayed quiet because that run "carries the tip".

Change

scripts/ci/nightly_next_stuck_run.sh: a new nightly-next tick step that runs before "Re-run a cancelled nightly-next tip". It lists the queued and in_progress nightly.yml runs on nightly-next (push or dispatch) and main's Promote nightly-next to ... dispatches, every page. A run is cancelled only when all of these hold:
- the run itself is queued or in_progress (never waiting, pending or requested) and has no pending deployment;
- one of its jobs has waited STUCK_MINUTES (20) queued with no runner, on a label that is not owned (glaeda-* is the owned-pool rescue's);
- every other job is finished or also unserved, so a run with a running job or a job waiting on a gate is never cancelled;
- a second read of the run, its deployments and its jobs, right before the cancel, still says so on the same attempt.
The rulebook allows cancelling a started nightly only when it is stuck or certain to fail; a code comment states why this counts as stuck. A newer tip is never a reason.
Fail-closed, against one schema. scripts/ci/nightly_next_shapes.jq (new) holds one schema per Actions object the tick reads (typed_run, typed_job, page_of), covering every field any of the scripts reads, each by its JSON type (a nullable field must still be present). gh fetches the raw JSON and jq checks each object against the schema before anything formats or compares it. An unreadable answer, or one of an unexpected shape, cancels, re-runs and dispatches nothing that tick.
Cancel, then force-cancel, each sent once. Both writes go out with no wrapper retries (GH_BUDGET_WAIT_ATTEMPTS=1), so a cancel never lands long after the last stuck read; a refused one leaves the run for the next tick. The step reads the run at most FORCE_AFTER + 1 (7) times, SETTLE_SECONDS (15) apart; only when the last read still shows the same attempt unfinished does it force-cancel, then read the same number of times again.
Only a real cancel counts. Nothing more is done unless the run is seen finished cancelled on the attempt judged stuck; a run that succeeded, failed or started a new attempt before the cancel took is left as it is.
The cancelled tip stays owed until a run of it exists. The stuck-run step only cancels. Building the tip again belongs to the next step, nightly_next_cancelled_tip.sh, which reads the obligation from GitHub itself every tick: the tip's newest nightly.yml run on nightly-next (push or dispatch) ended cancelled. GitHub writes that record with the cancel, and only a newer run of the tip clears it, so a refused re-run or dispatch, or a tick that dies after the cancel, is picked up by the next tick. Below MAX_ATTEMPTS (3) it re-runs that run through scripts/ci/nightly_next_rerun.sh, whose every read is fail-closed (a failed or malformed jobs, workflow or picker read sends nothing, not even the re-run of failed jobs) (new, taken out of nightly_next_cancelled_tip.sh): it reads every job left without a runner on any label, and when one of them takes its runner from a picker job's outputs and the re-run is safe (nightly_repick.py), it re-runs that picker so it chooses fresh; otherwise it re-runs the failed jobs, which evaluate their runs-on again. The picker is read from nightly.yml at the run's head (the job whose outputs those runs-on use as the label), never a fixed name: main's file calls it pick-nightly-app-runner, nightly-next's pick-nightly-build-runner. owned_pool_rescue.repick_job() takes the picker's key, defaulting to its old constant, so the owned-pool rescue is unchanged. At the cap it dispatches a fresh nightly.yml run on nightly-next (attempt 1, every runner picked afresh), Both writes call scripts/ci/nightly_next_tip_owed.sh (new) right before they go out: nightly-next still points at the tip, no run of the tip is unfinished in any state (an older run on a new attempt included), and the cancelled run is still the tip's newest. So a replacement is never doubled and only the current tip is built. Both rescue steps run only in ticks with no sha input, which share the one serialized concurrency group nightly-next-tick. A tick dispatched for one commit has a group of its own, so two of them could otherwise both find the same tip owed. The 5-minute backstop cron keeps an owed tip's wait under 5 minutes. Every write is sent once; a refused one fails the step and the next tick tries again.
- An older commit's run is cancelled and not re-run, because the tip builds. A promotion is not re-run, because the tick's own promotion request asks again.
The feed watch is unchanged here. #27179 already stops a run that never started (nothing running, every queued job 10 minutes without a runner) from holding the starvation dispatch back, and counts a run only when its head is the tip or contains it; this branch merged that and took main's feed watch whole.
nightly-next is read through the refs API, type-checked, in all three scripts: the tick's checkout keeps no credentials, so git ls-remote fails there (#27179). An unreadable answer sends nothing.

Stop condition and cleanup: the step's only waits are those bounded reads (allowlisted in the polling scan with that bound). Its scratch dir is removed on exit, and it runs continue-on-error, so the promotion request still runs.

Overlap: the general stall-aware runner failover (a job queued more than about 3 minutes on a label that has started nothing is moved onto a live member of the runner set) is separate work and is not duplicated here. This PR stays nightly-specific: it cancels and re-runs only nightly-next build and promotion runs, after 20 minutes. A job whose runs-on is a literal label goes back to that label on re-run; moving it is the failover's job.

A separate durable store (like #27114's OwedStore) is not needed: the record that the tip is owed is the cancelled run itself, which no tick can lose.

Not fixed here: the tick and the feed watch themselves run on Blacksmith Linux. #27022 adds CI_CONTROL_RUNNER to move them.

Tests

tests/test_nightly_next_stuck_run.py (rewritten, 21 tests, with a stateful fake GitHub; registered in tests/test-execution.toml and wired into CI fast guards). Covers: the 40771197361 shape is cancelled; running, gate-waiting, concurrency-pending, briefly queued, owned-label and served runs are kept; a run whose own status is waiting/pending, or that has a pending deployment, is kept; a run that changes between the reads and the cancel is kept; every failed read cancels nothing; force-cancel and its bounds; an unreadable status after the cancel never force-cancels; every run of the tip is handed to the cancelled-tip step; cancel writes go out once; a run that finished on its own before the cancel gets nothing more; an older commit's run is never re-run; 40 unfinished runs over several pages; tick step order.
tests/test_nightly_next_cancelled_tip.py: 20 pass, including a fresh run at the attempt cap, a refused fresh run staying owed until a replacement exists, no double dispatch and nothing sent on an unread list, including a Linux job that takes the pick (repick), jobs whose runner is not the pick's (failed jobs re-run), a renamed picker, and the checked-in nightly.yml naming a picker for its app build (so a rename can't silently turn the repick off).
Both scripts' test fakes answer the refs API and make git itself fail, so a script that still reads git fails its test.
Red on af017381: 19 failing cases across the six findings; red on f4554b14: 14 failing cases across the three follow-up findings; red on aadfbe69: the older-commit re-run and the picker name (4 cases); red on 6f758f2d: the stranded cancelled tip and retried cancels (11 cases); red on 5904051f: a moved tip and unexpected shapes (13 cases); red on e793323a: the feed watch's shapes (since replaced by main's feed watch); red on e34dca5c: a null run event (2 cases); red on db7f0edd: fail-open re-run reads, the older-run carrier and object-valued lists (11 cases); red on 1eb868f8: unchecked job name and runner_name (now the shared schema, tested field by field); red on bafe4102: missing nullable fields and ticks outside the serialized group; all pass on the head.
Also pass, after merging main at c34452d8: test_nightly_next_feed_watch (90), test_nightly_next_rolling (21), test_nightly_promotion_admission, test_ci_owned_pool_rescue, test_nightly_delivery_concurrency, test_ci_nightly_off_blacksmith, test_runner_label_policy, test_ci_fork_runner_routing, test_gh_budget_wait, test_ci_quota_hold_gate, the polling scan and its tests, the test registry validator, actionlint.

Observed / Not run

Observed: the tests above, run locally. Not run: a live cancel. Run 40771197361 itself has since been served.

Changelog

none

🤖 Generated with Claude Code

Agent session: agent session