acpmux: a host start ends on the host's answer or exit, never on a 10 s clock (P-AUSTIN B10)

LA
6d ago
+333-55

Summary

P-AUSTIN B10. In the Austin's-day A/A self-test (build 3ecd8020cb1, 5 rounds on cmux21, table), acpmux session/new failed with "agent host start did not finish within 10s" on 100 to 150 of 150 to 200 chat opens in every run, from the second 25-chat wave on. The hosts were alive. link::spawn ended every host start after a fixed 10 s (BOOTSTRAP_BUDGET), whatever the host was doing. That is a deadline that is "slower than N s" (R-L5) and a timing that decides correctness (R5).

After this change:

  • A host start ends on a signal, not a clock. link::spawn waits for the host's bootstrap answer, or for the bootstrap pipe to close. Before the host answers, only its exit closes the pipe, so a dead host fails at once and the error gives its exit status. If the caller drops the start (its own deadline, a closed chat), the start now ends the host before the pipes close. Before this change the host stayed running. BOOTSTRAP_BUDGET is removed. The callers keep their own deadlines: agent_start (90 s, configurable) for a chat, and the pool's START_BUDGET for a prewarm, which this PR also applies to the host start.
  • No start gate. An earlier head had a core-sized FIFO gate. Review B1 removed it: a start waits for a harness exec that other load starves, so the gate only cut a wave into rounds (on an 8-core Mac, a 25-chat wave would reach the 90 s agent_start deadline again). No measurement showed a gain.

Root cause of the slow starts (measured on the fleet, not fixed here)

Scratch probes ran on M4 Pro fleet Macs (14 cores, macOS 27): steps a54bf17e, 194b7954, 4f7275c9 on branch pa-b10-probe, which will not merge. In the probes, the acpmux part of a host start (exec, scope move, bootstrap frame, record fsync) took 1 to 2 s for 25 starts at once at load 100+, so it is not the cause. The cause is how the host starts its harness. On macOS, cpu_class.rs (#27600, which is in the measured build) applies PRIO_DARWIN_BG in pre_exec. That makes std use fork+exec, and the harness exec then runs at background priority. Command::spawn blocks until the exec finishes, and the host is single-threaded, so it sends its bootstrap answer only after the exec is done. 25 harness spawns started this way under CPU load took p50 14 s (max 17 s); under CPU plus disk load they took p50 28 s. The same spawns with no class, or with the posix_spawn QoS clamp (qos.rs), took 1 to 13 ms. This PR makes such a start slow but successful. The follow-up PR starts the harness with the posix_spawn background QoS clamp, so the class costs no slow exec (lead decision).

Testing

Focused command: cmux-ci run --class isolated --label macos-27 --script scripts/ci/cmux-tui-rust-check.sh --arg test --arg host_start_

  • Red at 6b0d44231dc (test commit only), step ee4745e6: 3 of 3 fail on their own assertions, each in less than 0.4 s. "25 of 25 slow-but-alive hosts failed: ... did not finish within 10s"; "a death decided by a clock"; "the host of a start its caller dropped ran on".
  • Green at 6496c10ab7b, step 8a0e09b0: 3 of 3 pass. Delta after review (gate removed, N2, N3) at 7ce811ce4fc: steps 5b915e25 (host_start_), 937f46c1 (agent_host), 11092d52 (clippy).
  • The tests use tokio's paused clock, so any time budget in the start path runs out before the stand-in host (tests/fake_slow_host.py) answers. No test waits on a real timer.
  • fmt, clippy (-D warnings) and the whole workspace on Linux, 4 shards at 6496c10ab7b: 803504e6, 0f1eb7b2, 81c397e8, 7ae548ad. fmt and clippy pass. Every acpmux test passes except the known base reds below.

Follow-ups (named, not in this PR)

  • N1: a dropped start ends the host, not a harness the host already forked (its own process group). The harness most likely ends on stdin EOF, and the base behaved the same on its 10 s timeout. Follow-up: on drop, end the harness group too.
  • N4: tests/agent_host.rs calls link::spawn with no bound now, so a host that never answers hangs those tests until the nextest timeout.

Observed / Not run

  • Observed base reds, not caused by this change:
    • All 7 cmux-tui::cli agents_chat_tools::* failures also fail at this PR's base 6b9c028ef33 (step a07d77a0, 5 of 5 and 4 of 9 there). They fail on #28147 too. Agent feat-tip-reds is fixing them on feat.
    • acpmux hermetic_daemon_guard (tests/web_image_attachments.rs, fix #28150), transport_path_coverage noun_first_path (#28119) and wireguard_ssh_credentials daemon_restart_sends_close_1012 (B3 is classifying it).
  • Not checked at the base, but this PR changes none of their files: cmux-agent-pane-policy::swift the_known_params_are_the_macos_hosts (policy.json against Swift, acp.session.fork live), agent_message_reply_in_place (reply.refused) and cmux-tui-core workspace_registry integrity_check_scope.
  • Merge of feat-cmux-next at 8fa71e25977 (two hunks resolved by hand: feat #28319's host_spec.orphan_exit line kept before this PR's START_BUDGET-wrapped spawn in hub/pool/start.rs, and orphan_exit: None added to the SpawnSpec in tests/host_start_wave.rs). Observed at 8fa71e25977: Linux fleet steps 9717c5fd (test host_start_), d9fc8621 (test agent_host), 2eebd4c7 (test orphan), 54943d4a (clippy -D warnings); results are added when they end. Not run at 8fa71e25977: macOS. This PR's own red and green (ee4745e6, 8a0e09b0, 5b915e25) ran on macos-27; the merge adds no macOS-only path (independent review), so Linux evidence is enough for the merge (lead decision 2026-10-30 19:2xZ).
  • Not run: the Austin's-day A/B. The harness owner runs it after the review passes. Until then this PR claims no perf change.

R9: R-L5 pass for the host start (it ends on the answer, the pipe end or a caller drop). R-L1, R-L2: no cap or pool in this PR. R-L3: known gap. Nothing in this PR refuses, but the callers' fixed deadlines remain: agent_start 90 s (configurable) and the pool's START_BUDGET 90 s; neither has a typed busy with retry_after_ms.
perf claim: none

Performance

No perf numbers: this PR changes when a start fails, not how fast it is. The Austin's-day A/B runs after the review; until then there is no perf claim.

Changelog

Fixed: Opening many agent chats at once on a busy Mac no longer fails with "agent host start did not finish within 10s".

Proof

No capture: the change is in the acpmux daemon and has no UI. The proof is the red and green fleet steps above.