ci(nightly-next): canary enforces after a clean report week; every pull starts the fix-forward

Merged
TE
17d ago
+901-31

PR 5 of the nightly release-train design (leo-workspace#624, section 1.6).

Problem

  • NIGHTLY_NEXT_CANARY is unset, so both canaries only report. A build that fails the canary still publishes and stays on the feed. Turning on enforce was a manual step nobody had scheduled.
  • The pre-publication canary runs once, so in enforce one flaky launch would hold the nightly back. The post-publication canary already takes two attempts.
  • A pull takes a bad build off the feed but can't move Macs that already have it. Only a higher build does that, and nothing starts one: the doc says to revert by hand.

What changes

Canary mode (scripts/ci/nightly_next_canary_mode.py; both canaries use it, so they always agree):

NIGHTLY_NEXT_CANARYMode
report / enforce / offUsed as given: this is the toggle.
unset or auto (default)report until NIGHTLY_NEXT_CANARY_ENFORCE_AFTER (default 2026-10-27 16:00 UTC, one week of report mode), then enforce, but only if the post-publication canary failed no build in the 7 days before that time (read from the nightly-next canary failed issue). A failure that week keeps auto in report until someone triages it and sets enforce or moves the date. The week is fixed, so the mode never flaps. If the read fails, the run stays in report.
anything elseFails the run, so a typo never skips the canary or enforces by accident.
  • nightly.yml verify-signed-nightly: resolves the mode (adds issues: read) and runs the canary twice before failing.
  • nightly-next-canary.yml: pick resolves the mode (sparse checkout of the script, issues: read). The pull and the issue text use the resolved mode.

Fix-forward (scripts/ci/nightly_next_fix_forward.py, new fix-forward job in nightly-next-rollback.yml, no environment, no secrets). It runs after every real pull, whether the canary or a person started it:

  1. maps the first pulled build and the build the feed now serves to commits (build number = <run id><attempt>, that run's head);
  2. names the suspect: the newest change in the first pulled build, meaning its PR, or its push for a direct push, with the owner. For a train commit, that is the second parent;
  3. if nightly-next-train.yml on NIGHTLY_NEXT_TRAIN_REF (default feat-cmux-next) declares a pulled_build input, dispatches it with only the inputs it declares, from this contract: pulled_build, good_build, quarantine (the suspect's 40-hex commit), reason;
  4. otherwise opens or comments on one nightly-next fix-forward needed issue with the compare range, the suspect, its owner and the manual fix-forward.

PRs 1 and 2 (durable verdict, planner, train workflow) aren't up yet. This PR plugs into them through the input contract above, and until the train exists it falls back to the issue.

Rollout notes

  • nightly-next-canary.yml is scheduled from main, so this file and scripts/ci/nightly_next_canary_mode.py also need a main mirror PR (the same shape as #21900). nightly.yml and the rollback workflow run from nightly-next, so promotion carries them. Until the mirror lands, the post-publication canary on main keeps the old behavior: it reads vars.NIGHTLY_NEXT_CANARY directly, so unset = report.
  • Leo's open call in the design ("enforce after one week of report mode") is the default here. Set NIGHTLY_NEXT_CANARY=report to never flip, or move NIGHTLY_NEXT_CANARY_ENFORCE_AFTER.

Verification

  • Implemented: unit tests tests/test_nightly_next_canary_mode.py (toggle, auto flip, clean or dirty report week, no flapping, read failure, CLI, workflow wiring) and tests/test_nightly_next_fix_forward.py (build โ†’ commit, train or direct-push suspect, dispatch with declared inputs only, issue fallback and comment, dry run). Both are wired into ci-guards (release-notary) and tests/test-execution.toml.
  • Verified: only by this PR's CI. Nothing ran on a local Mac.
  • Not exercised end to end: a real pull plus dispatch (it needs nightly-next and the train workflow).

๐Ÿค– Generated with Claude Code


View with [code]smith Autofix with [code]smith
Need help on this PR? Tag @codesmith-bot with what you need.


Summary by cubic

Adds auto-enforcement and the fix-forward to the nightly-next release train: the canary now flips from report to enforce after a clean report week, and every real pull starts a fix-forward instead of waiting on a manual revert.

  • A new scripts/ci/nightly_next_canary_mode.py resolver decides the mode for both canaries. report/enforce/off are used as given; unset or auto reports until NIGHTLY_NEXT_CANARY_ENFORCE_AFTER (default 2026-10-27 16:00 UTC) and then enforces, unless the canary failed a build in the week before that time. An unknown value fails the run.
  • The pre-publication canary uses the resolver and runs twice, matching the post-publication canary, so one flaky launch can't hold a nightly.
  • nightly-next-rollback.yml gains a fix-forward job that runs after every real pull. It dispatches nightly-next-train.yml on NIGHTLY_NEXT_TRAIN_REF with the pulled build's newest change quarantined when the train declares a pulled_build input; otherwise it opens or comments on one nightly-next fix-forward needed issue naming the change and its owner.