acpmux: a harness starts at normal CPU class and runs at background once its start is over; a late start says why

Merged
LA
6d ago
+852-175

Summary

Fixes the Chief outage on Lawrence's laptop, 2026-10-30 20:40 PDT (NIGHTLY acpmux a11272619). Every Chief turn failed with session/new: no answer within 60 s, and the daemon log said pooled session failed to start: no harness session within 90s harness=claude. #27600 starts every harness at PRIO_DARWIN_BG. The machine had load 125 on 18 cores and du/git scans on a 3.3 TB disk. The claude harness was already past its exec, but its main thread spent 90 s in SecTrustSettingsCopyTrustSettings -> open() / plist parse at disk tier 3, with under 1 s of CPU. Evidence: https://github.com/manaflow-ai/cmux/pull/28442#issuecomment-6538628785.

#28442 (open, P-AUSTIN B10) moves the background class from darwinbg before the exec to the posix_spawn QoS clamp. That fixes a slow exec, but the harness still runs at background I/O after the exec, so this hang stays. A clamp also cannot be removed from a running process. This PR changes the order instead: the harness starts at normal class, and the daemon lowers it after the start.

  1. A start never runs at background class. The harness starts at its starter's class (an agent host's utility clamp, or the daemon's class). Between fork and exec it clears a background class that it would inherit (cpu_class::start_at_normal, for example an agent host under ACPMUX_QOS=background). When ensure_child's start is over (initialize and session/new or session/load answered, including a pool claim and Claude's fork priming), the daemon lowers the running harness with setpriority(PRIO_DARWIN_PROCESS, pid, PRIO_DARWIN_BG) (ChildAgent::enter_run_class). Every process that the harness starts after that inherits the class, so the builds and tests of its turns stay below the GUI (#27600's goal). A pooled harness stays at normal class until a chat claims it. Model probes are never lowered (they never run a turn). The "a running process is never reniced" rule is changed on purpose: now the harness itself is reniced, once. A process that the harness started during its start (an MCP server) keeps its class, because acpmux does not list processes (check-no-process-scan.py). The class decision moved from SpawnSpec.cpu_class to the ChildAgent. The field is removed: serde ignores it from an older daemon, and a host is always the daemon's own binary.
  2. Retry. If a cold start fails or misses agent_start and its harness is at background class anyway (from a cause outside acpmux), the start runs one more time. The daemon also clears the second harness's external background flag right after the spawn, and logs agent_start_retried_at_normal (with the harness state) plus a warning.
  3. A late start says why. The agent_start and pooled-start deadline errors end with, for example, ; its agent process (pid 123) is alive at background CPU class (priority 4), ran 92 s, used 0.8 s of CPU, 0 of 14 threads running, or is not running / had not started. The text comes from proc_pidinfo of that one pid, with no process listing. data.harness carries the same fields. The text names no vendor.
  4. Chief waits past acpmux's deadline. Chief's turn session/new waited 60 s, which is less than acpmux's 90 s agent_start, so Chief never saw acpmux's reason. It now waits 100 s (SESSION_NEW_DEADLINE). Other control requests keep 60 s.

macOS facts used: setpriority(PRIO_DARWIN_PROCESS, pid, …) on another process of the same uid sets that task's external darwinbg flag (xnu do_background_proc: curp == targetp gives internal, otherwise external). A self call in pre_exec clears the internal flag. Darwinbg is task policy, so it survives exec and children forked later inherit it. Children that already exist are not changed. The behavior tests below check these on a fleet Mac.

Docs: cpu_class.rs module doc, docs/background-processes.md ("Start (macOS)"), config.rs cpuClass.

Owner: PL-acpmux (cpu_class.rs, agent.rs, agent_host); Chief (optchat-chief rpc/acpmux). Does not touch hub/idle_census.rs.

Testing

Behavior tests at the daemon's ACP client (an in-process daemon, serve_connection) and the agent host link, macOS (tests/cpu_class.rs, stand-in tests/fake_agent.py):

  • cpu_class_a_session_starts_at_normal_class_and_runs_at_background_once_started: the harness answers session/new above pri 4. After the answer it is at pri 4, and a tool that its turn starts is at pri 4.
  • cpu_class_a_profile_can_keep_its_harness_at_normal_class, cpu_class_a_profile_reference_that_resolves_to_normal_keeps_normal_class: the opt-out still works.
  • cpu_class_a_start_that_misses_its_deadline_says_what_its_harness_was_doing: a harness that cannot finish its start (FAKE_START_GATE) gives an agent_start error that names its pid, class, run time, CPU and threads. A normal-class start is not retried.
  • cpu_class_a_start_that_fails_at_background_class_runs_once_more: another process puts the first harness in background class during its start, and the harness fails initialize but stays alive. The start is retried once, logged with background: true, timedOut: false, and the session starts at normal class.
  • cpu_class_a_daemon_runs_a_hosted_harness_at_background_once_its_start_is_over: a real acpmux daemon run with agent hosts on (as in the app), driven by acpmux run "class-probe: …". The chat harness answers session/new above pri 4, then runs at pri 4, and a tool from its turn is at pri 4.
  • cpu_class_a_hosted_harness_starts_at_normal_under_a_background_host_then_runs_below: the agent host runs in background class (ACPMUX_QOS=background, pri 4). Its harness and a tool started during the start run above pri 4. After enter_run_class the harness and a tool its turn starts are at pri 4.

The test of a harness under load is the mechanism, not a timing: a start that has a deadline does not run at background class.

Focused command: cmux-ci run --class isolated --label macos-27 --script scripts/ci/cmux-tui-rust-check.sh --arg test --arg cpu_class

Results (fleet Macs, cmux-ci isolated, label macos-27; Linux on rbx):

  • Red at edf10ecc92b9 (tests only, on base 2b30971e45eb): step 898fcb69 on aws-m4pro-4, 3 failed, 4 passed. …runs_at_background_once_started failed with "the harness answered session/new at background class". The deadline test failed with the bare agent_start did not finish within 3s. The retry test failed with "the retried start failed: agent_start did not finish within 5s".
  • Green at 29be5565be7c: step d46d625f, 8 of 8 passed. macOS clippy step 0d8cc078: pass.
  • Review delta at c972b119bf6a (gate step 656de634, cmux10's Mac mini): 9 tests ran. All passed except the in-process hosted test: it read the background host at pri 46 once (a short kernel priority promotion), and it passed alone (FLAKY). Fixed in e83e364b1059: a class read is now the lowest of 5 reads, and the test ends its host before its checks. macOS clippy at c972b119 (gate step 8b3181ce): pass. rbx at c972b119: cargo clippy -p acpmux --all-targets -D warnings, plus hub_integration agent_start, session_pool and agent_host tests: pass.
  • Gate at e83e364b1059 (receipt /tmp/gates/e83e364b…v6.json, all 8 steps green):
    • macOS cpu_class, cmux-ci f68b6198 on aws-m4pro-3: 9 of 9 passed, including the real-daemon hosted test.
    • macOS clippy b72d87e4: pass. macOS never_answers_its_start 6668d34f: pass.
    • rbx Linux: acpmux clippy, hub_integration agent_start, session_pool (34 of 34), agent_host: pass.
    • Chief: rbx tests pass; macOS Chief tests (session filter) 71f8354c, clippy and fmt: pass.
    • Chief first-turn cost 47ee33a9: PASS, 18,680 of 50,000 tokens.
    • check-no-process-scan: ok.
  • Base reds, skipped or narrowed by name. Each one is red at feat-cmux-next 02e24e80c29c too, and the owner lane is acpmux-base-reds:
    • session_pool on macOS: cx-n1nrjk.
    • transfer_guard after_a_restart: cx-q0b1zh.
    • flow_trace_bind: cx-fel6hd.
    • peer_restart close_1012: cx-85z095.
    • setup keychain: cx-c8ouws.
    • startup_load: cx-ij2jg6.
    • subsessions: cx-b0utmu.
    • Chief acpmux_peer_moved: cx-2hwias.
    • Chief eval_upsell: cx-2ct95t.
    • Chief toolpack_mcp: cx-71g32y.
    • agent_host orphan_exit::a_daemon_connecting_while_the_harness_ends_is_told_it_exited: cx-o8vggt. It fails 17 of 20 runs alone at the base and 15 of 20 at the head.
  • Landing gate at the final rebased head: running (same 8 steps).

Observed / Not run

  • Not run: an A/B under real disk load on a laptop (load 125 + a 3.3 TB scan). The tests prove the mechanism (a start runs above pri 4; a running session is at pri 4), not a timing.
  • Not run: the full macOS workspace suite. Only the cpu_class and start filters, clippy, Chief's own tests and the Chief first-turn cost guard ran.
  • Not checked: the real agent of a launcher harness (a node shim, an ACP adapter that starts the CLI as a child). By design it stays at normal class (documented; follow-up).

Changelog

Fixed: On a busy Mac, starting an agent chat no longer waits behind other disk work, and a start that times out now says what the agent process was doing.

Proof

No capture: the change is in the acpmux daemon and Chief, and has no UI.