PR #27179 - manaflow-ai/cmux

Leo, 01:25Z: "and where are the nightlies!?"

What was wrong

The feed watch was red on every pass ("pushes have waited 1.9h, limit 0.5h"). Two parts of that are trigger timing, not build time:

Landing to build start. Since #26592 only the 30-minute tick (:07/:37) promotes, so a landing waits up to 30 minutes, plus the time until its tree is buildable, before its build starts.
The backstop could not fire. The feed watch runs at :07/:37 with a 25-minute trigger, so it acts 25 to 55 minutes after the last publication. It also counts any queued nightly run as active, so a run whose decide sat 34 or 73 minutes on a label that started nothing kept it quiet the whole time (cmuxterm-hq#4946 item 2).

The last 10 NEXT nightly runs on nightly-next (job timings from the Actions jobs API):

run	commit	landed (Z)	landing to run created	attempts	Mac Release build job	run created to publish-nightly done	landing to publish	longest queue
40802211271	a63986a4	10-29 01:25	19 min	1	8 min	29 min	48 min	2 min, legacy sign
40799674473	9cf2066a	10-29 01:05	13 min	1	13 min	34 min	48 min	3 min, deltas
40795807029	a21014c2	10-28 23:49	53 min	1	10 min	27 min	79 min	2 min
40789063286	73ff9f88	10-28 22:37	64 min	3	9 min	87 min	151 min	15 min, resolve-cmux-tui
40786303679	4e1a84f9	10-28 22:05	72 min	3	10 min	superseded	-	25 min, pick-build-runner
40778774768	67053da3	10-28 21:31	46 min	1	24 min	91 min	137 min	27 min, build inputs
40771197361	1f0f687b	10-28 21:15	5 min	3	10 min	127 min	132 min	35 min, decide
40752120475	14b20f80	10-28 18:54	18 min	2	16 min	failed	-	69 min, resolve-cmux-tui
40751861886	bb9ea920	10-28 18:49	20 min	1	-	cancelled	-	-
40738012059	3b5a2dd0	10-28 17:14	27 min	3	11 min	113 min	140 min	73 min, decide

The critical path of a clean run (40799674473): decide and inputs take 1 min, the Mac Release build 13 min (7 of them in xcodebuild), prepare 1 min, sign, notarize and staple the app 4 min, the DMG image on a Mac 5 min, sign and notarize the DMG 3 min, the two Mac verifies 7 min (6.5 of them downloading the signed app), and publish 1 min, for 34 min in all. The build-time cuts come in separate PRs. This PR covers the trigger.

What changes

nightly-next tick (nightly-next-tick.yml) now also runs when a feat-cmux-next cmux-next or cmux-tui artifacts push run finishes (that is when a landing becomes buildable), when a nightly-next build finishes, and on a 5-minute backstop cron. Before it promotes, scripts/ci/nightly_next_rolling.py applies the rolling rule:
- if nothing is building nightly-next, it promotes now;
- if a build is running, it promotes once the oldest landing that build does not carry has waited 10 minutes (vars.NIGHTLY_NEXT_ROLLING_MINUTES). A burst of landings becomes one build of the newest tip, started beside the running build. Nothing is cancelled, and publication still refuses a build no newer than the feed.
- A run whose publish-nightly finished, or that has sat 10 minutes with nothing running and every queued job without a runner, is not "building". A queued job's age comes from the first timestamp the API gives: its created_at, its started_at, then the run's run_started_at and created_at. With none of them it counts as old, never as zero age.
- Every read goes through the API with the job's token. nightly-next's head comes from the refs API and the oldest waiting landing from the compare API, so no git remote and no shallow history is involved. If nightly-next can't be read, the tick waits and logs a warning (the feed watch's dispatch for one commit skips the rule). A gap that can't be read counts as an old landing and promotes. Any other failed read promotes and logs a warning, because the promotion ignores a request that does not move nightly-next forward. A dispatch for one commit skips the rule.
- This is not the per-landing promotion that #26592 removed (six builds at once): the window allows about one new build per 10 minutes while one runs.
feed watch (nightly-next-feed-watch.yml, scripts/ci/nightly_next_feed_watch.py):
- runs every 10 minutes;
- dispatches cmux NEXT's tip once the feed or the tip is 20 minutes old (classic keeps its 25);
- stops counting a run that has not started as active (same test as above). An unreadable job list still keeps the run, so a failed read never doubles a build;
- a run holds the NEXT dispatch back only if it is a nightly-next build (or the train, when it is on) whose head_sha is the tip or contains it, per the compare API. Before, any active run created after the tip landed counted, including classic nightly runs on main. A failed compare does not count. The classic guard keeps its own rule.

Tests

The job and run payloads in the tests are recorded from a live nightly-next run (tests/fixtures/nightly_next_live_run_jobs.json, trimmed), including jobs with their times removed.
tests/test_nightly_next_rolling.py (new, 21 cases) covers: nothing building, so promote; inside the window, wait; window passed, build beside; a queued run that never started; a Mac job with no runner; a job with no times, aged from its run; a brief queue still counts; an environment gate is not stuck; a published run is done while its deltas run; fresh and stale promotions; nothing waiting; and the API reads (head from the refs API, gap from compare including a 400-commit gap, identical/behind/diverged/failed compare, unreadable nightly-next waits with a warning, other failed reads promote with a warning, no git remote used).
tests/test_nightly_next_feed_watch.py, 13 new cases: a main nightly beside a stale NEXT feed never holds the dispatch; a nightly-next run carries the tip only by its head (the tip, ahead or identical carry; behind, diverged, a failed compare or no head do not); an unstarted run does not block; one inside the grace period does; a job without its own times is aged from the run; no timestamp anywhere is old, not zero; running and gated jobs block; an unreadable job list keeps the run; a published run is not building while its deltas wait; a tip older than 20 minutes is built even when the feed is fresh; cmux NEXT uses 20 minutes; the watch cron is every 10 minutes. The two timestamp cases fail on the previous head.
The cadence assertions in tests/test_nightly_delivery_concurrency.py and tests/test_ci_nightly_off_blacksmith.py now pin the new triggers and require the rolling step to gate the request.
Run locally: all of the above, plus test_ci_workflow_run_sources.py, test_ci_quota_hold_gate.py, test_nightly_promotion_admission.py and test_nightly_next_cancelled_tip.py, pass.

Performance

metric	before	after	delta	harness or run link
landing to nightly run created, median of the last 10 runs	23 min (range 5-72)	at most 10 min under load, about 1-3 min on a quiet branch, by construction (filled in from the next 10 runs)	-13 min or more	Actions jobs API over nightly.yml runs on nightly-next (table above); the same script re-reads them after landing
feed-watch backstop: time from last publication until it acts	25-55 min	20-30 min	-5 to -25 min	cron arithmetic (:07/:37 + 25 versus every 10 min + 20)

Timers: no new loop. The new cron triggers run one bounded job each (the tick's 75-minute timeout, the watch's 5 minutes), and both keep their non-cancelling concurrency groups (one runs, the newest waits).

Overlap

#25943 (an idle lane, 10-27) had the 5-minute feed-watch idea. This PR covers it and the unstarted-run case.
#27095 changes the same active-run filter in the feed watch (stuck_run, a 20-minute rule for non-owned labels) and adds a tick step. The two compose: whichever lands second rebases a one-line conflict in starvation_dispatch.

Risk: more tick runs (each about 1 minute on a 2 vCPU Linux runner, one at a time), and up to about three nightly builds at once during a landing burst while builds take 30 minutes. Builds get faster in the follow-up PRs.

🤖 Generated with Claude Code

Agent session: agent session

Summary by CodeRabbit

Improvements
- Nightly preview builds are checked every 10 minutes, with a five-minute backstop and checks after relevant build events.
- Build requests respond to rolling wait thresholds, helping keep previews current when changes are waiting.
- Queued runs that have waited at least 10 minutes without a runner no longer hold back dispatch. Active builds carrying the latest changes continue to count.
- Dispatch decisions account for the age of both the feed and its latest changes, using a 20-minute threshold for the NEXT preview branch.