PR #22661 - manaflow-ai/cmux

PRs 1 and 2 of the nightly release-train design (leo-workspace#624, sections 1.1 to 1.5). This builds on #22599 (canary enforce mode and the fix-forward) and fills in the train that #22599 dispatches.

Problem

The nightly ships whatever commit last passed the Release compile. One bad commit blocks or breaks the whole feed, and nothing names who fixes it.
The canary's verdict is only an Actions cache key. It expires, and nothing can ask "which build was the last good one?"
The promote guard refuses anything that isn't a feat-cmux-next commit, so it can't ship "the tip minus the bad car".

What changes

Durable verdict (nightly-next-verdict.yml, new verdict job in nightly-next-canary.yml). Each scheduled canary check writes verdicts/<build>.json next to the cmux NEXT appcast in R2 (https://files-next.cmux.com/nightly-next/next/verdicts/<build>.json, not the legacy bare /nightly-next/ feed). The write runs on nightly-next in release-next. A pass counts only from a scheduled canary run whose canary <build> job succeeded, and the writer reads the build's commit from that build's own run.

Planner (scripts/ci/nightly_next_train.py):

base: the newest feed build with a pass verdict. With no verdicts yet it falls back to nightly-next's head.
plan: the cars of base..tip, first parent. There is one car per squash-merged PR, per direct push (repository activity) and per merge, each with its owner. Merged PRs come from one listing, 100 a page, so a day of cars takes a few API calls rather than one per PR.
candidate: a two-parent train commit (nightly-next head, tip) whose tree is the tip minus the quarantined cars. A revert that doesn't apply also pulls out the later cars that touch the same files ("quarantined with pr-N").
verify: rebuilds that tree from the commit's Nightly-Train-Revert: trailers. This is G1'.

Train (scripts/ci/nightly_next_train_drive.py, nightly-next-train.yml): the per-car guard hook for PR 3, then the candidate's Release compile on a nightly-train/* ref. On red it bisects k = 4 ways, quarantines the culprit and rebuilds. It gives up after 3 rounds or 120 minutes, ships nothing and keeps the current nightly. On green it asks main to promote. Each quarantined car gets one comment, on its PR or in the nightly-next quarantine issue. A shipped train's quarantine carries into the next train, so a fixed car rejoins on its own. The workflow accepts #22599's fix-forward inputs (pulled_build, good_build, quarantine, reason).

Promote guard (promote-nightly-next.mjs, nightly.yml promote and ack steps):

G1': the second parent is on feat-cmux-next, and the tree is verified by main's own script.
G2': a green nightly-next train Release compile from a nightly-train/* dispatch run of this exact SHA.
The first parent must be the current nightly-next.
G3/G4 are unchanged, and a plain commit still takes G1/G2.

Defaults and toggles (Leo's open calls in the design):

Variable	Default
`NIGHTLY_NEXT_TRAIN`	unset = `dry-run`: every tick plans and the run summary is the plan; promotion stays as it is today. `on` swaps the tick's promotion for the train. `off` disables it.
`NIGHTLY_NEXT_TRAIN_K` / `_ROUNDS` / `_CAP_MINUTES`	4 / 3 / 120

Rollout notes

Most of this runs from main: the tick, the canary schedule, the promote job and guard, and the dispatch registration of the two new workflows. So a main mirror PR follows, the same shape as #21900 and #22599. Until that lands, nothing here changes behavior.
on also needs a new nightly-next-train environment (deployment branch feat-cmux-next) holding the release App key. A train commit changes workflow files relative to nightly-next, and GITHUB_TOKEN can't push those. That is Leo's credential call, so the default stays dry-run.
The two new workflows are added to .github/release-surface.txt: the train workflow is now a promotion gate, and the verdict workflow holds the R2 token.

Verification

Implemented and tested: tests/test_nightly_next_train.py has 30 cases. They use real git in a scratch repo for cars, candidates, conflict pull-ins, carried quarantine and the G1' tree check (including a forged revert list). Fakes cover the bisection loop (one culprit in 40 cars within 3 rounds of 4, two culprits, round and time caps, no result), verdict trust, the last-good base and the workflow wiring. tests/test_nightly_promotion_admission.py gains 10 G1'/G2' cases. Both are wired into ci-guards (release-notary) and tests/test-execution.toml.
Ran locally: only these Python and node unit tests, plus actionlint on the changed workflows. Nothing was built.
Not exercised end to end: a real train run, which needs the main mirror and the environment.

Not in this PR

The unsigned in-app smoke on the candidate. It gets only the Release compile; verify-signed-nightly still runs on the signed app.
Flake triage after two failures of the same guard, and the 3-trains-out digest.
Per-car guards (PR 3, a7) and the pulled-build banner (PR 4, a4) plug in through NIGHTLY_NEXT_TRAIN_CAR_GUARD and the verdict documents.

🤖 Generated with Claude Code

Agent session: cosmic-gray

^{Need help on this PR? Tag @codesmith-bot with what you need.}

Summary by cubic

Implements the nightly release train for cmux NEXT: instead of shipping whatever commit last passed the Release compile, the nightly now ships the newest build with a passing canary verdict, minus any broken cars that get quarantined on the way.

What changed

The canary writes a durable verdict (verdicts/<build>.json) next to the appcast in R2, so the last good build survives cache expiry. A pass only counts from a scheduled canary run whose canary <build> job succeeded and whose own verdict pass <build> job ran; the verdict dispatch warns instead of failing until nightly-next carries the verdict workflow.
The planner computes one car per squash-merged PR, direct push, or merge between the last good build and tip, then builds a candidate train commit (tip minus quarantined cars). A revert that conflicts pulls in later cars touching the same files; carried cars re-read their labels so nightly-hold persists. The G1' tree check refuses any revert outside the tip's history.
The train driver compiles the candidate on a nightly-train/* ref on hosted macos-26, never Blacksmith (NIGHTLY_NEXT_TRAIN_RUNNER stays as an override) and, on red, bisects k = 4 ways with 3 rounds / 120 minutes cap, quarantines the culprit, and rebuilds. Giving up ships nothing and keeps the current nightly.
The promote guard gains G1' (second parent on feat-cmux-next, tree rebuilt and verified by nightly_next_train.py) and G2' (a green train Release compile for that exact SHA); the first parent must be the current nightly-next. A plain commit still takes G1/G2.
NIGHTLY_NEXT_TRAIN defaults to unset, meaning dry-run: every tick plans and logs the plan, promotion stays as today. on swaps the tick's promotion for the train; off disables it. The train is one-way once a train commit ships, so keep on.

Rollout

Everything runs from main (tick, canary schedule, promote job, dispatch registration), so nothing changes behavior until the main mirror PR lands.
Turning on the train needs a new nightly-next-train environment (deployment branch feat-cmux-next) holding the release App key, because a train commit changes workflow files and GITHUB_TOKEN cannot push those.
The two new workflows are registered in .github/release-surface.txt. Covered by 30 new train tests using real git in a scratch repo and 10 new G1'/G2' promote-guard cases, all wired into ci-guards (release-notary).