PR #32328 - manaflow-ai/cmux

The ask, from the nightly freshness lane: "each build starts about 25 min after the landing it carries ... start each tip's build when it lands, not after a batch window, and build alongside the running one."

What puts the time between a landing and its build

I measured the 30 nightly-next push runs from 09:11Z to 13:10Z on 2026-11-02, plus run creation for 380 landings since 05:00Z. On an ordinary tick, a landing starts building in about 5 minutes:

part	typical	source
landing to tick promotion request (a source run finishes or the 5-min cron fires, then the tick runs about 1.5 min)	2.7-4 min	tick runs, promote runs 41450611602, 41448773525, 41449659742
promotion run on main moves nightly-next, and the push run appears	0.3-0.4 min	same
run created to `build-nightly-app` start (decide, inputs, picker)	1.2-2.3 min (median 1.6)	jobs of the 30 runs
batching of older landings into the newest tip (rolling window, `NIGHTLY_NEXT_ROLLING_MINUTES`=5)	+3-5 min for the oldest landing a run carries	oldest-carried vs tip

The 25-30 min cases all came from one cause. All ticks share the nightly-next-tick group, so they run one at a time. Their housekeeping steps read through gh_budget_wait.sh. When a read was refused, that wrapper backed off 60+120+240+300 s. The notice went nowhere because Python captures the subprocess's stderr. No promotion could start until the tick finished:

tick	held	step
41392901357	13.2 min	rolling rule 12.1 min
41398820119	14.2 min	stuck-run rescue 12.1 min, then "could not be read"
41433152543	22.5 min	rolling rule 20.1 min
41440726202	25.7 min	stuck-run rescue 15.2 min (hit its step timeout), rolling rule 9.0 min

Here is what that did to two runs:

d887a9ec1b52 landed 11:09:17Z and its tree was ready at 11:11:40Z, but the promotion came at 11:23:19Z. Its build started 15.7 min after the tip landed and 28.3 min after the oldest landing it carries.
7036ae37b772 landed 12:00:02Z, but the promotion came at 12:22:16Z. Its build started 23.9 min after the tip landed and 30.0 min after the oldest landing.

Change

In nightly-next-tick.yml, three steps now set GH_BUDGET_WAIT_ATTEMPTS: "1": the stuck-run rescue, the cancelled-tip re-run and the rolling rule. A refused read in them now fails at once into the fallback each step already has:

the rescue cancels nothing;
the re-run step fails, and it has continue-on-error;
the rolling rule promotes, or waits when nightly-next itself can't be read, as its docstring says.

The next tick (a landing, a finished build, or the 5-minute cron) reads again. Only the promotion request still waits through a rate limit. Writes in these scripts already went out with GH_BUDGET_WAIT_ATTEMPTS=1. The rolling rule is unchanged, so a newer tip still starts beside a running build and never cancels it. Publication stays latest-wins, and no concurrency setting changes.

This targets main because a workflow_run or schedule tick runs from the default branch. Every recent change to this file (#29924, #30172, #30259) landed on main.

Performance

metric	before	after	delta	harness or run link
tick time held by rate-limit backoff, 05:15-13:10Z on 2026-11-02	4 ticks, 13-26 min each (75 min total)	0 (a refused read fails in seconds)	-75 min	ticks 41392901357, 41398820119, 41433152543, 41440726202
landing to `build-nightly-app` start, p95 (380 landings)	23.8 min	~16.8 min (projected: excess hung-tick time removed from the 28 affected landings)	~-7 min	nightly-next push runs 41418619409..41450667345, jobs API
landing to build start, max	30.0 min	~10 min for the same landings (tick 1.5 min + promote + build start)	~-20 min	runs 41436492688, 41444184160
landing to build start, median	7.7 min	7.3 min	-0.4 min	same

The "after" figures are projections from the same data. The next 24 hours of ticks will confirm them.

Risk

Low. Only the tick changes, in three steps that already have continue-on-error. When GitHub refuses a read, the rescue or the re-run now waits for the next tick, a few minutes later, instead of waiting inside this one. The rolling rule may promote on a read failure, which it already does today after its waits. Main's promote job still refuses a request that does not move nightly-next forward. No secrets, permissions, runners or concurrency groups change. Prior art: #27179 and #27248 (the rolling rule, unchanged here), #27272 (the rescue), and the open #30509, which changes this file's runs-on only.

Follow-ups

These are not in this PR:

The rolling window still adds 3-5 min for older landings. Setting NIGHTLY_NEXT_ROLLING_MINUTES to 0 would start a build per landing, about 50 an hour today. That needs Mac capacity sized first.
gh_budget_wait.sh could also log every wait to the step summary, so a wait under captured stderr shows up.

Tests

tests/test_nightly_delivery_concurrency.py: all 15 functions, called directly
tests/test_gh_budget_wait.py (18), test_nightly_next_rolling.py (30), test_nightly_next_stuck_run.py (24), test_nightly_next_cancelled_tip.py (15), test_nightly_next_feed_watch.py (94), test_ci_workflow_run_api_budget.py (46), test_ci_quota_hold_gate.py, test_ci_cloud_overflow_switch.py, test_ci_nightly_off_blacksmith.py, test_ci_fork_runner_routing.py, test_delivery_workflows_cancel.py, test_ci_admission_placement.py, test_ci_workflow_run_sources.py, test_nightly_promotion_admission.py: all pass
bash tests/test_nightly_universal_build.sh: PASS
bash tests/test_ci_self_hosted_guard.sh: PASS
actionlint 1.7.7 on nightly-next-tick.yml: clean

Agent session: agent session
🤖 Generated with Claude Code

^{Need help on this PR? Tag @codesmith-bot with what you need.}

Summary by cubic

Fixes nightly-next builds starting 15-30 minutes after a landing instead of roughly 5 minutes, by stopping the tick's housekeeping steps from waiting out GitHub rate limits.

The stuck-run rescue, cancelled-tip re-run, and rolling rule now set GH_BUDGET_WAIT_ATTEMPTS: "1", so a refused read fails instantly into each step's existing fallback instead of backing off 60+120+240+300 s invisibly (Python captured the wrapper's stderr) and holding the serialized tick group.
Only the promotion request still waits through a rate limit. The next tick — a landing, a finished build, or the 5-minute cron — retries the read.