ci(cmux-next): run the base Mac tier at most once every 15 minutes, on the tip
Leo: "u know im looking at the state of ci on the [fleet dashboard] and idk do we REALLY need to be compiling/testing THAT often...?"
This is fix 1 of the CI waste research, manaflow-ai/cmuxterm-hq#5424 ("CI waste: compiles and tests that add no signal"; finding: the base tier runs on every landing). On the window that research measured, feat-cmux-next took 781 landings, and 648 of them touched a cmux-next push path. Each of those pushes ran the whole base Mac tier, and most of that work was for commits that were already superseded.
What changes
- A push no longer runs the base Mac tier. On a
feat-cmux-nextpush, path routing turns off three jobs:cmux-next daemon tests,cmux-next generated filesandcmux app scheme compile (Debug)(scripts/ci/cmux_next_base_tier.py defer).- Still on every push: both Release compiles (nightly-next's input), the DEV landing compile, the fast tier,
checks, web and the rest. - When
vars.CMUX_NEXT_DEV_LANDING_COMPILE=0, the push's scheme compile doubles as the DEV landing build, so the push keeps it.
- Still on every push: both Release compiles (nightly-next's input), the DEV landing compile, the fast tier,
- A new kick,
cmux-next-base-tier-kick.yml, runs the tier instead.- It starts on every push, on Linux, through
vars.LINUX_RUNNER. - Its workflow concurrency group has
cancel-in-progress: false. One kick runs while the newest pending one waits, and older pending kicks, which never started, are replaced. - It waits once until 15 minutes after the last base tier run, then reads the tip. If a cmux-next push path changed since the last base tier commit, it dispatches
cmux-next.ymlwithbase_tier=true. - That run is named
cmux-next base tier · <sha>and runs exactly those three jobs, plus their Linux companions: crash ends the app and push attribution. It always checks generated files on thecmux-next generated filesjob itself, never folded into daemon-test's build, so a pull request's generated-files red always has a base verdict under the same job name.
- It starts on every push, on Linux, through
- A base tier run whose cmux-tui tree isn't published yet leaves a tree-wait marker with
tip_suite=base, as the tip daemon suite does.cmux_next_tree_notify.pythen dispatches the same-tree run on that exact commit, even after the tip has moved on. - Off switch: setting
vars.CMUX_NEXT_BASE_TIER_COALESCE=0puts the tier back on every push and turns the kick off. Both read the same variable.
Never cancelled, delayed or coalesced: any nightly, DEV or promote build, or the Release compile. This PR adds no cancel-in-progress: true and no evicting group to those workflows.
Performance
Window: 2026-10-29T20:00Z to 2026-10-30T20:00Z, the same 24 h the research used.
Harness: replay.py (kept in the lane's scratch). It walks the feat-cmux-next first-parent landings from git and matches each landing's changed files against cmux-next.yml's push paths. It then replays the kick exactly as cmux_next_base_tier.kick behaves: dispatch at max(landing, last + 15 min) + about 80 s kick overhead, at the tip at that moment, only if a push path changed. Before and after use the same Mac hours per tier run, taken from the research's Actions measurement of the window. Job times come from glaeda's per-job records on the owned minis.
| metric | before | after | delta | harness or run link |
|---|---|---|---|---|
| base tier runs triggered per day | 648 (one per relevant push) | 95 | -553 (-85%) | replay.py |
| Mac hours/day, base tier (daemon tests + generated files + scheme compile) | 93.1 (measured) | 58.7 | -34.4 h/day | research measurement × replay.py |
| landing → its base tier dispatch, p50 / p90 / max | 0 (dispatched at once) | 7.3 / 13.8 / 15.0 min | +15 min worst case | replay.py |
| landing → base tier verdict, typical (wait + route + p50 slowest job) | ~15 min (job only) | ~22 min | +7 min | replay.py, glaeda records |
| landing → base tier verdict, worst (max wait + route + p90 slowest job) | ~21 min | ~36 min | +15 min | replay.py, glaeda records |
The "before" hours are lower than the research's 86 h/day estimate. That figure covered every base tier in the research, including Release, DEV, nightly, media, base autofix and tip test-compile. This PR coalesces only the three that feed no nightly, DEV or promote producer. The after-hours are an upper bound. Before, superseded pushes often never got a runner, while the kick's run is always the tip's, so the saving per run is at least what the table shows.
What the data does not cover: the glaeda records come from the owned minis, without the reserved hosts, AWS Macs and Blacksmith macOS. The run counts come from git and are exact; the hours use the research's Actions-wide measurement.
Why no signal is lost
- Every landing is still covered. It gets a base tier run within 15 minutes plus about 80 s, at a tip that contains it. Each dispatch takes the tip at the moment it fires, and a newer pending kick replaces an older one only before that one started. So no relevant landing goes uncovered.
- A red points to a range of PRs.
push-attributionnow also runs for a red base tier run (it needs every base tier job, the crash gate included), and it starts the culprit range at the commit the last green run tested. That commit is the sha in the run's name, because a same-tree run'shead_shais the newer tip. The range is the landings since the last green base tier run, about 7 in a busy 15 minutes and at most about 10 in the window. Push attribution comments on each PR when there are 5 or fewer. - Bisecting a red: push a branch at a commit in the range and dispatch
cmux-next.ymlon it (gh workflow run cmux-next.yml --ref <branch>), which runs every tier. Halve the range until one PR is left. This is written up indocs/ci/cmux-next-tiers.md. For Swift package suites,scripts/ci/package_bisect.pyapplies unchanged. - Pull requests are unchanged. Every routed tier still runs on a PR, so most breaks are still caught before landing. The base tier only backstops what slips through.
Readers of base runs (audit)
Most landings now have no base tier run at their own sha, so each reader must fall back to the nearest run that has one and say so when it is stale.
| reader | finds a base verdict? | change |
|---|---|---|
scripts/ci/base_red_excuse.py (and test_reds.py, which it runs) | Yes. Suite titles now map to the jobs each suite can run (SUITES). cmux-next base tier · carries daemon tests, generated files, the scheme compile and crash-ends-app. In the base-run window and in the dispatch fallback, runs of a suite that never runs the job (DEV daily, tip Swift suite) are skipped unread, so the 40-job-list budget reaches the base tier run. The audit line names the tested commit and the run's time ("the newest feat-cmux-next verdict for it"). | Tests: test_the_base_tier_run_is_the_base_verdict_for_its_jobs, test_suites_that_never_run_the_job_are_skipped_unread_in_the_window, test_a_green_base_tier_run_still_blocks. They fail on base and pass here. |
leo-workspace merge_guard.py run_excuse | Yes, unchanged. For cmux, a failing Actions job with no base verdict of its own goes to base_red_excuse.py, which reads base tier runs (above). | none |
| HQ base-red-watch / next-autoland | Partly. Their push-run passes read event=push runs, which no longer carry these three jobs. They see the tier only through the base-red excuse. | A follow-up HQ PR makes both read cmux-next base tier · dispatch runs. Until it lands, a red base tier shows up through push attribution and base_red_excuse, not as a base-red-watch alert. |
| NOW.md base-reds section | Derived from base-red-watch, so it follows the HQ change above. | follow-up HQ PR |
scripts/ci/next_batch.py base fallback | Yes. When the base probe gives no result and the base's own push run has no verdict for a job, the batch takes that job's outright red from the newest of the last 6 base tier runs that ran it to a verdict, so a run that skipped it (tree not yet published) never hides an older red. It logs that run's commit and creation time as "the newest for it, not necessarily the base head". | Tests: test_without_a_probe_the_newest_base_tier_run_stands_in_for_its_jobs, test_a_base_tier_run_that_skipped_a_job_never_hides_an_older_red (both fail on base). |
scripts/ci/cmux_next_push_attribution.py | Yes. It scans push,workflow_dispatch and keys each run by the commit in its name. | Test: test_a_base_tier_run_starts_the_range_at_the_commit_it_tested (fails on base). |
scripts/ci/cmux_next_tree_notify.py | Yes. A tip_suite=base marker (dispatch on feat-cmux-next only, tiers within daemon and scheme) is current whatever the tip is, and its same-tree run gets base_tier=true, so it keeps its name. | Tests: test_a_base_tier_runs_its_exact_commit_after_the_tip_moves, test_only_a_dispatch_on_feat_cmux_next_is_a_base_tier. |
| main's tip suite dispatcher (Swift, daemon) | Unchanged. Its daemon leg now overlaps the base tier and is partly redundant; it is left alone here. | none |
dev-daily dispatch / dev-daily publisher / dev_release_product_reuse.py | They read the scheme compile name only on DEV daily runs (cmux NEXT DEV daily · ), which this PR does not touch. | none |
dogfood-artifact-publish.yml | Reads PR runs only. | none |
Dependency: base autofix exports (fix 2)
The base autofix's pushed-exports pack (fix 2, another PR) is produced in the push's generated-files job, which now runs in the base tier dispatch. Its owner is adapting the producer and lookup to accept the cmux-next base tier · <sha> dispatch, take the sha from the title, and wait about 40 min. Until both land, the autofix's Mac job regenerates the exports as it does today, so nothing breaks; that saving simply waits for fix 2.
Incremental builds and the shared cache
This PR changes no build. The three jobs keep their runners, their per-runner kept DerivedData on the minis (incremental), and their R2 web-bundle and compile-cache restores. Coalescing helps the incremental cache: one run every 15 minutes builds the diff of about 7 landings on a warm runner, instead of 7 partly overlapping builds spread across hosts. The shared SSD compile cache is not wired into these Debug jobs today. That is a separate change, and coalescing neither blocks nor needs it.
Size
About 900 changed lines, not split: the kick, the push routing that defers the three jobs, and the readers that must find a base verdict on a dispatch run (base-red excuse, next_batch fallback, push attribution, tree notify) have to land together. Landing the deferral without the readers would leave base reds unexcusable and unattributed; landing the readers alone changes nothing. About a third of the lines are tests.
Observed / Not run
Observed:
- New tests (
tests/test_cmux_next_base_tier.py, registered intests/test-execution.tomland run in cmux-next.yml's Tier routing step): kick timing and window, path relevance, the coalesce switch, the route step run under bash with a stub router, the dispatch branch, and the kick workflow's concurrency and permissions. The added excuse, attribution, tree-notify and next_batch tests fail on the base code and pass on this branch. - Every test module that reads
.github/workflowswas run on this branch. The ones that fail also fail the same way on base:- test_check_ghostty_zig_workflows (collection)
- test_ci_fork_runner_routing (only the line number in its message moved)
- test_ci_upload_artifact_retention
- test_ci_workflow_run_sources
- test_cmux_next_tour
- test_cmux_tui_publisher_testing
- test_next_batch's 5
land_flag/did_not_landtests
- The invariant suites pass: no cancel-in-progress on producers (
test_nightly_delivery_concurrency,test_ci_pr_runs_cancel_superseded,test_ci_side_lane_placement), runs-on with no pins or hosted labels (test_ci_runs_on_values,test_ci_runs_on_fallback,test_runner_label_policy),test_ci_required_checks_are_bounded,test_ci_workflow_path_filter_parityandtest_cmux_next_dev_kick. - actionlint 1.7.7 reports the same two findings as base on cmux-next.yml: the 10-input dispatch limit, which base already exceeds, and one existing shellcheck note. The new workflow lints clean.
scripts/cmux-next/prepush-gate.py: no new violations.
Not run:
- A live base tier dispatch. The kick needs this workflow on feat-cmux-next, so the first real kick runs after merge; the off switch is the variable above.
- Blacksmith and AWS Mac hours are not in the per-job records (see Performance).
risk:normal (CI routing only; no product code)
Agent session: agent session
🤖 Generated with Claude Code