ci(nightly): nightly-next runs stop skipping their own builds; no 32 vCPU pins; display visual gate opt-in

Merged
TE
6d ago
+56-23

Leo, 2026-10-31: "i dont like doing display tests anymore". Leo, 2026-10-31: "FOR OUR NIGHTLIES ARE THEY END TO END ON OUR FLEET? WE CAN'T BE RELYING ON BLACKSMITH THAT CUTS OUT". Rulebook: "Nightly freshness" and "No pins".

What was wrong

The NEXT nightly fell more than 100 landings behind (NOW.md, 04:35Z). Every nightly-next run after f8c1ccbed (published 04:14Z) crawled. Blacksmith's 32 vCPU size waited 8 to 15 minutes per job from about 03:45Z: the overflow probe at 03:55Z had 32 vCPU start after 11 minutes and 16 vCPU after 3 s. 17 nightly jobs named blacksmith-32vcpu-ubuntu-2404 literally:

  • both pickers (pick-nightly-build-runner, pick-nightly-sign-runner)
  • signer-app, signer-app-notary and signer-dmg
  • every publish job

The run chains about ten of these one after another, so each stall added up. The display Mac visual gate (visual-gate-nightly) also took 37 minutes of f8c1ccbed's critical path.

The main cause: every run skipped its own build

In resolve-nightly-cmux-tui-client, "Leave the build slot to a newer nightly-next push" sets superseded=true whenever nightly-next has moved past the run's commit. Promotions came about every 6 minutes. A run took 12 minutes or more of Linux queue to reach that step. So every push run from 04:28Z to 05:23Z found itself superseded and built nothing (runs 41170916165 and 41171540012 log "nightly-next moved from ... the newer push's run builds it"), and no run ever built. The step dates from #21917, when pushes shared one build slot. NEXT runs are run-scoped now, and the rulebook says a newer tip builds alongside a running one. publish-nightly already publishes latest-wins against the feed (publish-recheck-recovered.sh, --superseded-ok), so builds running side by side are safe. The skip is now opt-in: NIGHTLY_NEXT_SUPERSEDE_SKIP=1.

What this changes

  • Those 17 jobs take the expression the rest of the nightly's Linux jobs already use: vars.LINUX_RUNNER == 'blacksmith-16vcpu-ubuntu-2404' && 'blacksmith-16vcpu-ubuntu-2404' || 'blacksmith-32vcpu-ubuntu-2404'. It only ever yields one of two literal Blacksmith labels, never the variable's value. So a job that holds a secret still cannot follow a variable onto a reused machine (signing rule 7).

  • route-visual-gate-nightly runs only when NIGHTLY_NEXT_VISUAL_GATE names a mode. Unset now means off, so no nightly waits on a display Mac. Setting the variable to report, enforce or auto turns the gate back on.

  • The signing guard (tests/test_ci_signing_keychain_is_deleted_before_bundle_code.py, rule 7) now accepts vars.X == 'L' && 'L' || when L is an ephemeral label. The variable is only compared, and the job runs on the literal. A self-check covers the accepted form and the two forms it still refuses. The guard's 4 remaining failures (sign-nightly-identity-legacy and generate-nightly-deltas reading CI_NIGHTLY_NEXT_BUILD_RUNNER and CI_SIDE_LANE_RUNNER) are on base since a7b6a611, in lines this PR leaves alone.

  • tests/test_nightly_legacy_publish_linux.py expects the new expression.

Done by hand at the same time (mitigation)

  • LINUX_RUNNER = blacksmith-16vcpu-ubuntu-2404 (04:50Z).
  • NIGHTLY_NEXT_VISUAL_GATE = off (05:02Z).

Risk

CI routing only, no product code. If the 16 vCPU size stalls instead, setting LINUX_RUNNER back moves every nightly Linux job together. Not in this PR, as follow-ups: fleet-first Linux routing for the nightly (no GitHub runner on our own Linux hosts exists today) and a CI test for it.

šŸ¤– Generated with Claude Code