#27248 ci(nightly-next): the rolling rule starts at most one build per window, the newest tip

Leo, 01:25Z: "and where are the nightlies!?"

What broke

The rolling rule from #27179 promotes beside a running build once the oldest waiting landing is 10 minutes old. That age is measured from landing. When the cmux-tui tree stalls promotion (no waiting commit has a published tree for 20+ minutes), every landing is already past the window by the time its tree publishes, so each tick promotes again:

tick	time (Z)	decision (from its log)
40817598158	04:25:3x	promote 185716ef: "bdfc3dee has waited 21 min"
40817892088	04:28:13	promote 68e51e81, beside 185716ef's build started 2 min before
40817979989	04:29:13	"nothing is building", 4 s after the promotion that moved nightly-next, before its run existed; promote 913e9372
40818228571	04:31:41	promote again, "83871c91 has waited 17 min"

At 05:14Z five nightly-next builds were in flight: 40817728360, 40818080811, 40818586159, 40819439227 and 40821132202. They queued 3-5 min on Blacksmith macOS; one build ran 18 min on a cold mini and one picker job took 4.9 min. Only the newest build's publication matters, and publication refuses a build no newer than the feed.

What changes

scripts/ci/nightly_next_rolling.py:

• One start per window. While builds run, a new build starts only once the newest start is also a window old. A start is a nightly-next push run, or a promotion created in the last 2 minutes (its run may not be listed yet). The request still walks from the tip, so the one start is always the newest buildable commit. Older waiting tips are never built separately.
• Race guard. A promotion created in the last 2 minutes counts as building whatever its status reads, so "nothing is building" can no longer be read in the gap between the branch move and the push run.
• One listing. The script reads nightly-next's recent push runs once (event=push, 20 runs), and takes the head's runs and the newest start from that listing.
• Unchanged:
  • nothing building → promote at once;
  • a dispatch for one commit (a person, or the feed watch's starvation guard) skips the rule;
  • nothing is ever cancelled;
  • publication is still newest-wins.

Tests

tests/test_nightly_next_rolling.py::OneStartPerWindowTests (7 cases):

• a start inside the window holds an old landing;
• once the newest start is a window old, the newest tip starts beside it;
• another commit's newer start still counts;
• a promotion seconds old counts before its run exists;
• with nothing building and no recent start it promotes at once;
• a replay of the 04:25-04:36 ticks against the runs that existed then gives exactly one start (04:25:35) and the next at 04:36:20, where before it gave 04:25, 04:28 and 04:29;
• evaluate reads one listing.

All 7 fail on main (6 errors, 1 failure) and pass here. The 21 existing cases pass after aligning the fake's run to the head commit (the old code filtered by head_sha in the API call; the fake ignored that filter). test_nightly_next_feed_watch.py, test_nightly_delivery_concurrency.py and test_ci_nightly_off_blacksmith.py pass.

Performance

metric	before	after	delta	harness or run link
builds started 04:25-04:35Z (newest-tip promotions after the tree stall)	3 (185716ef, 68e51e81, 913e9372) plus 442ed021 at 04:35	1 (185716ef), then the newest tip at 04:36	-2 to -3 starts	replay test over ticks 40817598158, 40817657120, 40817892088, 40817979989 and 40818228571
nightly-next builds in flight at once, worst observed	5 (05:14Z)	at most about build time / window (35/10, so ~3-4)	about -1 to -2	Actions runs API, nightly.yml on nightly-next

Land-to-publish will be measured from the next 10 runs after merge, with the same harness as the table in #27179.

No new timer or loop.

Risk: a newer tip can wait up to one window (10 min) behind the newest start instead of starting at once. That is the intended trade: one build of the newest tip, not one per tree publication.

🤖 Generated with Claude Code

Agent session: agent session

---

^{Need help on this PR? Tag @codesmith-bot with what you need.}

---

Summary by cubic

The nightly-next rolling rule now starts at most one build per 10-minute window, and that build is always the newest tip, so a tree stall no longer triggers a burst of parallel builds.

Previously, the rule only required the oldest waiting landing to be a window old. After a cmux-tui tree stall, every landing was already past the window, so each tick promoted again (three starts in four minutes and five builds in flight at once). Now the newest build start must also be a window old. A start is a nightly-next push run, or a promotion created in the last 2 minutes whose run may not be listed yet; a still-running promotion counts as a start for the full window, and a read failure after the run listing waits rather than promoting beside a young start.

Nothing is cancelled; if nothing is building and no start is recent, it still promotes at once, and single-commit dispatches skip the rule.

• Reads nightly-next runs in one listing instead of two calls.
• Adds 9 test cases, including a replay of the 04:25 burst that now yields one start.

Summary by CodeRabbit

• Bug Fixes
  • Nightly build promotions now account for recent build starts and promotions, including promotions whose associated build run has not appeared yet.
  • Builds wait when the newest start is within the configured interval; once it has passed, the existing waiting-commit age check determines whether to promote.
  • If a later status check fails while a recent start is known, the build waits rather than promoting.