cmux-next: the fast tier runs as a build-fleet client when every macOS label is full or dead

Merged
TE
4d ago
+745-23

Builds on #28249 (merged as 38c2c9aed1d). This branch is this PR's own 7 commits rebased onto current feat-cmux-next; none of #28249's commits remain in its diff.

What

When every owned and Blacksmith macOS label the fast tier could take is full or dead, the tier no longer waits: it runs as a thin cmux-ci client on the build-fleet gateway, and cmux-next fast tier keeps its check name and exit code.

#28249 added fleet_jobs to the placement (filtered to FLEET_CLIENT_JOBS, which it left unset, and read only through label_full_or_dead) and reserved the gateway outputs without wiring any job to them. This PR wires the fast tier with the client + report pattern swift-canary-fleet / swift-canary already use:

  • fast-tier-fleet (new, name cmux-next fast tier (build fleet)): runs on needs.macos-placement.outputs.gateway_runner only when fleet_jobs lists fast-tier and placement saw an idle gateway in this attempt. permissions: {}, no checkout, no action, no secret. One step: cmux-ci run --class light --script scripts/ci/package-test-lane.sh --ref <REF> --arg=suite ... for each package of fast-tier.json's Swift entries (CmuxHomeRender: MotionTokenTests, IntentMappingTests; CmuxNext: ModelMatrixTests, PaneZoomMotionTests, CEFProfileStorageLogTests, AgentSubagentPanesFocusTests), in parallel, --no-supersede. REF is the exact commit fast-tier itself checks out: github.sha, a pull request's merge commit (as swift-canary-fleet does), so a head that breaks against the current base is still caught. On a push it is the pushed sha. It then reads cmux-ci status <step> and requires every ref in the record (and source_digest) to equal REF. Output verdict is pass, fail or unavailable. The controller's record decides: pass needs state done with exit_code 0 (and a client exit 0). A step that ended with a nonzero exit is a fail even if the client exited 0. Step ids are read as 6 to 64 hex, as the fleet tools read them. Fail, no verdict, or a step at another commit all fail the job; only a pass at REF exits 0.
  • fast-tier: needs fast-tier-fleet. When that job ran (success, failure or cancelled), it runs on blacksmith-2vcpu-ubuntu-2404 (4 vCPU under CI_CLOUD_OVERFLOW_SAVED, as swift-canary does) with a 5 minute timeout. Its first step reports the client's verdict (pass + success: exit 0; fail: exit 1; anything else: exit 1 with "no verdict"), and every other step is off (CMUX_FAST_TIER_FLEET). Its runs-on no longer names gateway_runner. When the client job skipped, nothing changes.
  • Push route (with #28260): fast-tier-fleet honours needs.path_route.outputs.fast_tier on pushes as well as pull requests, so a push whose diff since the last finished fast tier reaches no Mac tier skips the client as it skips the tier. test_the_client_runs_only_when_placement_listed_its_tier_with_an_idle_gateway checks the two gates agree on both events (red without the gate).
  • macos-placement: FLEET_CLIENT_JOBS: fast-tier in the place step. next_macos_cascade.py is unchanged.
  • Gateway room (side_lane_placement.fleet_gateway()): the client runs on gateway_runner beside GATEWAY_JOBS (app-compile-fleet, swift-canary-fleet, release-compile-fast), and the gateway is judged for those alone. Placement now keeps fleet_jobs only while the gateway has an idle runner for each listed job beyond GATEWAY_JOBS and the run's own jobs (counted as gateway_route() counts them). Otherwise it lists nothing and fast-tier keeps its route, rather than its client queueing behind another gateway job until its timeout. A re-run emits no gateway, so it lists none. Tests: GatewayRoute.test_a_fleet_client_needs_its_own_idle_gateway_runner, test_main_lists_a_fleet_job_only_with_a_gateway_runner_for_its_client (caught when the call in main() is removed), test_a_rerun_lists_no_fleet_job. No label is hard-coded for the gateway; gateway_runner comes from placement.
  • No whole job on the gateway: #28249's FleetGateway.test_gateway_output_is_reserved_without_wiring_raw_mac_jobs becomes test_no_tier_runs_on_the_gateway_itself: no tier's runs-on names fleet_jobs or gateway_runner, or resolves to the gateway when placement lists it. Its check that FLEET_CLIENT_JOBS is unset is dropped, since this PR sets it.

Tiers with and without a client

  • fast-tier: client added. Its Swift entries map one to one to package-test-lane.sh suite filters, the allowlisted light script. One difference: fast-tier.py runs a failed entry once more to label it FLAKY; suite mode fails on the first failure. Both are red, so the gate is the same.
  • swift-test: no client. It is not a cascade job (CASCADE_JOBS is fast-tier, swift-canary, cmux-scheme-compile, release-compile), so place() never leaves it unplaced and fleet_jobs can never list it. Its route-selected suites and side steps need a checkout to plan.
  • daemon-test: no client. It is not a cascade job either, and it needs the same-tree cmux-tui daemon. A fleet suite step without that daemon reports "no result", so a client could never honestly pass.

.github/runners.json classifies cmux-next.yml#fast-tier-fleet as apple-swift-tests, beside the other gateway clients (the routing table reads the gateway label as macOS).

Gateway label allowlist

The client job's id is fast-tier-fleet and its name is cmux-next fast tier (build fleet). The glaeda gateway allowlist for cmux-next.yml needs this id (the reporting job fast-tier never runs on the gateway).

Guard tests

GatewayRunsOnlyTheFleetClient in tests/test_cmux_next_checks_workflow.py:

  • Every job whose runs-on can reach gateway_runner through fleet_jobs has runs-on equal to the gateway output, permissions: {}, no container, services, environment or secrets, and only run steps. Every command word is the client or a shell builtin (a shlex-based parse), with no absolute path and no eval, exec, source, command, builtin or env.
  • In report mode every fast-tier step but the report is off, checked by evaluating each step's if with every earlier step skipped.
  • The client step is run with PATH empty against a fake cmux-ci. It passes only on a pass at REF, and fails with the right verdict for: a suite failed, a step at another commit, a queue timeout, no worker, no client, and a REF that is not a full sha. Any non-builtin it ran would show as "command not found".
  • FLEET_CLIENT_JOBS and the <tier>-fleet jobs match one to one. The tier never names gateway_runner, reports on Linux only when the client ran, and has every other step off. The report step's exit matches the client's.
  • The client's suites equal fast-tier.json's gating Swift entries by package.
  • BuildFleetGatewayJobsFallBack.FALLBACKS now includes fast-tier-fleet ā†’ fast-tier.

FleetClientJobs in tests/test_ci_next_macos_cascade.py: with every label full or dead, FLEET_CLIENT_JOBS puts exactly the listed jobs in fleet_jobs (written fast-tier). Nothing is listed when no client is configured, when a Blacksmith label is under its cap, when the side label has a live listener, or when there is no snapshot. The workflow lists only cascade jobs that have their <job>-fleet.

Red/green: with these test files on #28249's earlier head (d3289ff720a), GatewayRunsOnlyTheFleetClient + BuildFleetGatewayJobsFallBack fail (7 failures, 12 errors) and FleetClientJobs fails (1 failure, 1 error). On this branch they pass. On an earlier #28249 head (be77727b0ce), whose runs-on sent whole Mac jobs to the gateway, test_a_client_job_runs_nothing_but_the_client failed for each of those seven jobs. Two mutations of this branch are also caught: an ungated Run the fast tier step, and a git fetch added to the client step.

Same-host runs, this branch vs its base (330eebb6fd5): test_cmux_next_checks_workflow.py 173 OK (base 165 OK), test_ci_next_macos_cascade.py 125 OK (base 121 OK), test_ci_side_lane_placement.py 81 OK (base 78 OK). test_ci_runs_on_fallback.py, test_cmux_next_checkout_retry.py, test_ci_macos_xcode_selection.py, test_ci_fork_runner_routing.py and test_ci_fast_tier_anchor.py pass on both. test_ci_change_areas.py fails the same one test (test_repository_helpers_route_by_where_they_run) on both. test_ci_self_hosted_guard.sh and actionlint give the same output on both. Every test file that ci-guards.yml or the cmux-next CI groups run (502 files naming cmux-next.yml, placement, cmux-ci or runners.json) gives the same result by name on this branch and on its base.

Risk

Low. Behaviour changes only when placement lists fast-tier in fleet_jobs, which needs the placement script from the base (after #28249) and every macOS label full or dead. With no idle gateway the client skips and fast-tier takes its usual route. No new permission, secret or runner label. The client job holds no token.

Agent session: agent session

šŸ¤– Generated with Claude Code