ci: move CLI regressions to CLI product tests and rebalance the seven app-host shards
Summary
The app-host unit test shards were the last lane for most heads, and the slowest shard set the verdict. Measured from the last 24 hours of step timings (p50):
| shard | batches | strict steps and setup | content |
|---|---|---|---|
| 1 | 269 s | none | ~270 s |
| 2 | ~185 s | none | ~185 s |
| 3 | ~190 s | none | ~190 s |
| 4 | ~110 s | shell and CLI no-socket lanes ~116 s, bun, nushell, brew bash | ~230 s |
| 5 | ~160 s | nine focused gates ~140 s | ~300 s |
| 6 | ~125 s | seven focused gates ~290 s | ~410 s |
| 7 | ~180 s | global search ~28 s | ~210 s |
Shard 6 ran p50 414 s on Blacksmith and 590 s on the minis, while shard 3 finished in 266 to 294 s.
Changes (same seven workers, so no extra gui slots):
- CLI regressions move to
CLI product tests. The shell and CLI no-socket lanes, the bash prompt bootstrap regression, and the terminal creation CLI regression only need the CLI binary, which that job already restores. They take node, Bun, the pinned nushell and the ghostty submodule with them, and each runs after an earlier failure so one failing lane does not hide the others. The bundled theme picker helper step is dropped: underCMUX_SKIP_ZIG_BUILD=1it always skipped, and nightly.yml still runs it against a Zig-built helper. That job ran p50 83 s on Blacksmith and 152 s on the minis; with the move it measured 168 s on Blacksmith. - The focused regression shard is split. Agent notification semantics (with its node and Bun setup) owns worker 4; Cloud machine ordering shares worker 7 with global search. Worker 6 keeps My Devices, Copy Mode, five-tab memory, Pi Feed and the tmux detach gates (~128 s).
- Reservations
4=85 5=140 6=128 7=99, from the step medians. - Timings refreshed from runs 36335158751, 36350164643 and 36346707374. The old file summed to 1824 s against about 1260 s of observed batch wall and underweighted some shards by half; the new one sums to 903 s of test time, and each batch adds about 22 s of launch, the same on every worker.
Expected effect
The planner predicts about 238 s of test work per worker, with every batch non-empty, down from about 410 s on shard 6. With setup that is about 5 min per shard, down from up to 7 on Blacksmith and 10 on the minis. Getting to 4 min would take nine workers; an earlier revision of this PR did that (Blacksmith shards measured 220 to 324 s), but it needs two more gui slots per full run, and gui slots are the scarcer resource right now.
Coverage change: the CLI lanes now run with CLI product tests, which runs for CLI changes and full suites. Before, they ran on every all-shard run, including the unit-ci label without a full suite; now they also run for CLI-only changes.
Known failures not from this change
The three seen on an earlier run are fixed on main and merged in here: Cloud machine ordering (#15204), feedToolResultDoesNotReopenSettledCompletion (#15173), and the zsh watcher fallback test's PATH (#15237).
Testing
tests/test_ci_cmux_unit_test_shard.py(balance within 10% of the mean),tests/test_ci_change_areas.py,tests/test_ci_test_execution_registry.py,tests/test_ci_cli_product_routing.py,tests/test_ci_notification_semantics.py, registry validator, actionlint.- The layout change runs every shard on this PR, so CI shows the new balance.
🤖 Generated with Claude Code