ci(nightly-next): canary enforces after a clean report week; every pull starts the fix-forward
PR 5 of the nightly release-train design (leo-workspace#624, section 1.6).
Problem
NIGHTLY_NEXT_CANARYis unset, so both canaries only report. A build that fails the canary still publishes and stays on the feed. Turning onenforcewas a manual step nobody had scheduled.- The pre-publication canary runs once, so in
enforceone flaky launch would hold the nightly back. The post-publication canary already takes two attempts. - A pull takes a bad build off the feed but can't move Macs that already have it. Only a higher build does that, and nothing starts one: the doc says to revert by hand.
What changes
Canary mode (scripts/ci/nightly_next_canary_mode.py; both canaries use it, so they always agree):
NIGHTLY_NEXT_CANARY | Mode |
|---|---|
report / enforce / off | Used as given: this is the toggle. |
unset or auto (default) | report until NIGHTLY_NEXT_CANARY_ENFORCE_AFTER (default 2026-10-27 16:00 UTC, one week of report mode), then enforce, but only if the post-publication canary failed no build in the 7 days before that time (read from the nightly-next canary failed issue). A failure that week keeps auto in report until someone triages it and sets enforce or moves the date. The week is fixed, so the mode never flaps. If the read fails, the run stays in report. |
| anything else | Fails the run, so a typo never skips the canary or enforces by accident. |
nightly.ymlverify-signed-nightly: resolves the mode (addsissues: read) and runs the canary twice before failing.nightly-next-canary.yml:pickresolves the mode (sparse checkout of the script,issues: read). The pull and the issue text use the resolved mode.
Fix-forward (scripts/ci/nightly_next_fix_forward.py, new fix-forward job in nightly-next-rollback.yml, no environment, no secrets). It runs after every real pull, whether the canary or a person started it:
- maps the first pulled build and the build the feed now serves to commits (build number =
<run id><attempt>, that run's head); - names the suspect: the newest change in the first pulled build, meaning its PR, or its push for a direct push, with the owner. For a train commit, that is the second parent;
- if
nightly-next-train.ymlonNIGHTLY_NEXT_TRAIN_REF(defaultfeat-cmux-next) declares apulled_buildinput, dispatches it with only the inputs it declares, from this contract:pulled_build,good_build,quarantine(the suspect's 40-hex commit),reason; - otherwise opens or comments on one
nightly-next fix-forward neededissue with the compare range, the suspect, its owner and the manual fix-forward.
PRs 1 and 2 (durable verdict, planner, train workflow) aren't up yet. This PR plugs into them through the input contract above, and until the train exists it falls back to the issue.
Rollout notes
nightly-next-canary.ymlis scheduled from main, so this file andscripts/ci/nightly_next_canary_mode.pyalso need a main mirror PR (the same shape as #21900).nightly.ymland the rollback workflow run fromnightly-next, so promotion carries them. Until the mirror lands, the post-publication canary on main keeps the old behavior: it readsvars.NIGHTLY_NEXT_CANARYdirectly, so unset = report.- Leo's open call in the design ("enforce after one week of report mode") is the default here. Set
NIGHTLY_NEXT_CANARY=reportto never flip, or moveNIGHTLY_NEXT_CANARY_ENFORCE_AFTER.
Verification
- Implemented: unit tests
tests/test_nightly_next_canary_mode.py(toggle, auto flip, clean or dirty report week, no flapping, read failure, CLI, workflow wiring) andtests/test_nightly_next_fix_forward.py(build โ commit, train or direct-push suspect, dispatch with declared inputs only, issue fallback and comment, dry run). Both are wired intoci-guards(release-notary) andtests/test-execution.toml. - Verified: only by this PR's CI. Nothing ran on a local Mac.
- Not exercised end to end: a real pull plus dispatch (it needs
nightly-nextand the train workflow).
๐ค Generated with Claude Code
Need help on this PR? Tag @codesmith-bot with what you need.
Summary by cubic
Adds auto-enforcement and the fix-forward to the nightly-next release train: the canary now flips from report to enforce after a clean report week, and every real pull starts a fix-forward instead of waiting on a manual revert.
- A new
scripts/ci/nightly_next_canary_mode.pyresolver decides the mode for both canaries.report/enforce/offare used as given; unset orautoreports untilNIGHTLY_NEXT_CANARY_ENFORCE_AFTER(default 2026-10-27 16:00 UTC) and then enforces, unless the canary failed a build in the week before that time. An unknown value fails the run. - The pre-publication canary uses the resolver and runs twice, matching the post-publication canary, so one flaky launch can't hold a nightly.
nightly-next-rollback.ymlgains afix-forwardjob that runs after every real pull. It dispatchesnightly-next-train.ymlonNIGHTLY_NEXT_TRAIN_REFwith the pulled build's newest change quarantined when the train declares apulled_buildinput; otherwise it opens or comments on onenightly-next fix-forward neededissue naming the change and its owner.