ci(nightly-next): pushes share one build slot; the newest push builds

Merged
TE
18d ago
+143-9

Every push to nightly-next built its own head. Pushes queued for the trusted macOS runner, and each intermediate head got a full build: on 2026-10-18, three pushes in three minutes started three 90-minute builds. In the week to 2026-10-19, merged → first published nightly-next was the slowest stage of the whole PR path: p50 5.8 h and p90 2.1 d, against 0.8 h for open → green and 0.1 h for green → merged at p50.

What changes:

  • One build slot for nightly-next pushes. build-nightly-app gets a job-level concurrency group, nightly-next-build-app, with cancel-in-progress: false. At most one build waits, and a newer push replaces it. A running build always finishes. The workflow-level group stays run-scoped, so the post-publication checks (macOS 15 launch, legacy identity) still can't hold the next build. main, rc, schedules and dispatches keep run-scoped slots.
  • A superseded run never takes the slot. Job concurrency is decided when a job becomes ready, not in push order. So an older run whose resolver finished late could replace a newer run's waiting build. The last step of resolve-nightly-cmux-tui-client now re-reads nightly-next and sets superseded=true when the ref has moved. The four app-build jobs then skip (build-nightly-app, the ghostty helper, prepare, sign).
  • The rescue never reruns an old head into the slot. owned_pool_rescue.py used to see "no newer run" for nightly-next, because those runs are never pending at the workflow level. It now treats a run whose head is no longer nightly-next's tip as moved.
  • Only a true slot replacement goes unreported. report-nightly-failure drops build-nightly-app (cancelled) only when that job never got a runner and ran no steps. A timed-out or hand-cancelled build still reports, and so does every other failed job in the run.

Changelog

none

Validation

  • Red, then green, twice:
    • python3 -m unittest tests.test_nightly_trusted_routing (12 tests): commits 1 and 3 are the red tests, commits 2 and 4 the fixes. The tests evaluate the slot group for nightly-next push, dispatch and main; the build if: with superseded true and false on both macOS build jobs; and the reporter's runner check.
    • tests.test_ci_owned_pool_rescue.Nightly covers "a nightly-next run whose head is no longer the tip is never rerun". It failed before the fix and passes after.
  • tests.test_nightly_next_recovery passes.
  • tests.test_ci_owned_pool_rescue: 133 of 135 pass. The 2 SideLanes failures fail identically on feat-cmux-next.
  • python3 scripts/verify-local.py --affected origin/feat-cmux-next: 9 of 10 pass. localization fails identically on the base (34327 pre-existing parity errors).
  • A review subagent read the first version and found the problems the next commits fix: wrong SHA in the superseded check, push-order inversion, rescue reruns, and timeouts hidden.
  • A second review of the repair found that the reporter lacked actions: read, so its job lookup would have thrown and reported nothing. The reporter now has it. A failed lookup still reports. A build cancelled while queued for a runner, with nightly-next not moved, also still reports.
  • Not run locally: actionlint (CI's workflow lint covers it).

Check after merge, on the first nightly-next run whose build waits for the slot:

  • What the jobs API says for a job held by concurrency. If it reports queued with trusted labels rather than pending, the rescue would count it as stuck and needs to skip it.
  • Whether a run whose build was replaced still publishes a recovered build through find-nightly-next-recovery. If it does, gate recovery on the build not being cancelled.
  • The nightly.yml runs on nightly-next should show at most one build-nightly-app waiting, and no new nightly-next-failure comment for a replaced one.

šŸ¤– Generated with Claude Code