ci(nightly-next): pushes share one build slot; the newest push builds
Merged
TE
18d ago+143+143+012345678901234567890123456789-9-9-0123456789
Every push to nightly-next built its own head. Pushes queued for the trusted macOS runner, and each intermediate head got a full build: on 2026-10-18, three pushes in three minutes started three 90-minute builds. In the week to 2026-10-19, merged ā first published nightly-next was the slowest stage of the whole PR path: p50 5.8 h and p90 2.1 d, against 0.8 h for open ā green and 0.1 h for green ā merged at p50.
What changes:
- One build slot for nightly-next pushes.
build-nightly-appgets a job-level concurrency group,nightly-next-build-app, withcancel-in-progress: false. At most one build waits, and a newer push replaces it. A running build always finishes. The workflow-level group stays run-scoped, so the post-publication checks (macOS 15 launch, legacy identity) still can't hold the next build. main, rc, schedules and dispatches keep run-scoped slots. - A superseded run never takes the slot. Job concurrency is decided when a job becomes ready, not in push order. So an older run whose resolver finished late could replace a newer run's waiting build. The last step of
resolve-nightly-cmux-tui-clientnow re-reads nightly-next and setssuperseded=truewhen the ref has moved. The four app-build jobs then skip (build-nightly-app, the ghostty helper, prepare, sign). - The rescue never reruns an old head into the slot.
owned_pool_rescue.pyused to see "no newer run" for nightly-next, because those runs are never pending at the workflow level. It now treats a run whose head is no longer nightly-next's tip as moved. - Only a true slot replacement goes unreported.
report-nightly-failuredropsbuild-nightly-app (cancelled)only when that job never got a runner and ran no steps. A timed-out or hand-cancelled build still reports, and so does every other failed job in the run.
Changelog
none
Validation
- Red, then green, twice:
python3 -m unittest tests.test_nightly_trusted_routing(12 tests): commits 1 and 3 are the red tests, commits 2 and 4 the fixes. The tests evaluate the slot group for nightly-next push, dispatch and main; the buildif:withsupersededtrue and false on both macOS build jobs; and the reporter's runner check.tests.test_ci_owned_pool_rescue.Nightlycovers "a nightly-next run whose head is no longer the tip is never rerun". It failed before the fix and passes after.
tests.test_nightly_next_recoverypasses.tests.test_ci_owned_pool_rescue: 133 of 135 pass. The 2SideLanesfailures fail identically on feat-cmux-next.python3 scripts/verify-local.py --affected origin/feat-cmux-next: 9 of 10 pass.localizationfails identically on the base (34327 pre-existing parity errors).- A review subagent read the first version and found the problems the next commits fix: wrong SHA in the superseded check, push-order inversion, rescue reruns, and timeouts hidden.
- A second review of the repair found that the reporter lacked
actions: read, so its job lookup would have thrown and reported nothing. The reporter now has it. A failed lookup still reports. A build cancelled while queued for a runner, with nightly-next not moved, also still reports. - Not run locally: actionlint (CI's workflow lint covers it).
Check after merge, on the first nightly-next run whose build waits for the slot:
- What the jobs API says for a job held by concurrency. If it reports
queuedwith trusted labels rather thanpending, the rescue would count it as stuck and needs to skip it. - Whether a run whose build was replaced still publishes a recovered build through
find-nightly-next-recovery. If it does, gate recovery on the build not being cancelled. - The nightly.yml runs on nightly-next should show at most one
build-nightly-appwaiting, and no newnightly-next-failurecomment for a replaced one.
š¤ Generated with Claude Code