acpmux: start a background harness under the posix_spawn QoS clamp, not darwinbg before its exec (P-AUSTIN B10)

LA
3d ago
+435-28

Summary

P-AUSTIN B10, the root cause behind #28412. On macOS a background-class harness (the default since #27600) was lowered with PRIO_DARWIN_BG in a pre_exec hook, so its exec itself ran throttled. Command::spawn returns only after that exec, and the agent host answers its controller only after the spawn returns. On busy M4 Pro fleet Macs (14 cores, macOS 27), 25 harness spawns at once took p50 14 s (max 17 s) under CPU load and p50 28 s under CPU and disk load. The same spawns took 1 to 13 ms unthrottled or under the posix_spawn QoS clamp (scratch probe steps 4f7275c9, a54bf17e on the never-merge branch pa-b10-probe). In the Austin's-day self-test that became "agent host start did not finish within 10s" on 100 to 150 of 150 to 200 chat opens.

After this change, a background harness starts under the posix_spawn QoS clamp at background (posix_spawnattr_set_qos_clamp_np, POSIX_SPAWN_PROC_CLAMP_BACKGROUND). This is the setter that qos.rs already uses for agent hosts, and it is the call behind taskpolicy -c background. The clamp applies at the exec, so the exec runs at the starter's class. The harness and every process it starts run at background priority (pri 4), as before, so #27600's drag-lag result stays. The pre_exec hook execs the harness itself: a posix_spawn with POSIX_SPAWN_SETEXEC (same pid, descriptors, process group and folder as the execvp it replaces) of the program resolved on the harness's PATH, its argv and its complete environment, all built before the fork. A missing program still fails the spawn with ENOENT, so program_wait still works. A macOS without the clamp setter falls back to darwinbg. Both start paths use it: the agent host (spec.env) and a direct child (ACPMUX_AGENT_HOSTS=0; harness_env keeps a remote profile's cleared environment).

Why the clamp is applied in the forked child, not by a posix_spawn in the parent

The lead asked whether the parent can call posix_spawn with the clamp directly, as qos.rs does. It cannot without a larger change, for three reasons:

  1. The cwd pin. A sub-session harness execs only in the folder its start checked (cwd_pin.rs): a pre_exec hook stats . after the chdir and fails the spawn with ESTALE if the folder was swapped. posix_spawn has no point between its chdir and its exec where that check can run. (posix_spawn_file_actions_addfchdir_np on an fd opened at the check could replace it. That is a design change of the pin, owned by PL-acpmux.)
  2. The child handle. The host (host.rs) and ChildAgent hold a tokio::process::Child: tokio-registered pipes, reaping through tokio's SIGCHLD driver, kill_on_drop, wait. tokio cannot build a Child from a pid that another call spawned. A parent-side posix_spawn needs its own child type with its own pipes and reaper on both paths.
  3. std's setup. std's fork path also resets the signal mask and SIGPIPE and makes the other descriptors close at exec. A direct posix_spawn must repeat each of these (SETSIGMASK, SETSIGDEF, CLOEXEC_DEFAULT).

Safety of the call in the child. std would call execvp at this point; the hook calls posix_spawn with SETEXEC instead, with every argument built before the fork. libsystem's posix_spawn (xnu libsyscall/wrappers/spawn/posix_spawn.c) fills a descriptor on the stack and makes the __posix_spawn syscall: no allocation, no lock. Its execve wrapper does the same, and both consult only a filter hook that is NULL on customer installs. So the hook does nothing that std's own execve does not do. Apple does not document posix_spawn as async-signal-safe, and that is the residual risk the reviewer judges. The alternative is items 1 to 3 above.

Review delta (FIX at f605bb4c7fb, fixed at 9a051797210)

  • Review 1 (security): harness_env no longer reads the profile's env a second time. harness_command and harness_env both take the remote fact from remote_environment(profile): the raw ACPMUX_REMOTE_ENV_POLICY key, which acpmux sets to the literal 1 after it removes any profile value (hub/spawn.rs). There is no secret lookup, and a value that is a reference counts as remote (fail closed). Red: a_remote_harness_env_stays_cleared_when_its_secret_is_gone_after_the_build (f985c420201).
  • Review 2: the SAFETY comment states that Apple does not document posix_spawn as async-signal-safe, which xnu source was read, and why it is accepted.
  • Review 3: ENOEXEC (a script with no #!) runs with /bin/sh, as execvp does. Red: cpu_class_a_background_harness_script_without_a_shebang_starts_with_sh (586d4d73d30). A program found under a relative PATH folder is exec'd by its relative path, from the folder the child changed into after the cwd pin's check.
  • Review 5: one POSIX_SPAWN_SETEXEC (in qos.rs); CpuClass::Background's doc is fixed.
  • Review 6: the darwinbg fallback logs once.
  • Not changed (named here): review 4, "call apply last" is a comment, not enforced (both call sites are correct); review 3 minors: resolve accepts any file with an x bit (execvp would skip an EACCES entry and go on), and a non-UTF-8 env value set on the command is changed by to_string_lossy.

Owner: PL-acpmux (cpu_class.rs, qos.rs, agent_host/host.rs, agent.rs, agent/command.rs). The lead tells the owner.

Testing

Focused command: cmux-ci run --class isolated --label macos-27 --script scripts/ci/cmux-tui-rust-check.sh --arg test --arg cpu_class

  • The red tests (cpu_class_a_{hosted,direct}_background_harness_is_clamped_not_darwinbg_at_its_exec) start a background harness, hosted and direct. They require the harness and a tool it starts at pri 4, and a process the harness starts not darwinbg (getpriority(PRIO_DARWIN_PROCESS, 0) == 0). The check is on the mechanism, not on timing.
  • Delta at 9a051797210: red steps 154ae6a3 and 2bcdd85b at 586d4d73d30; green d9e2e455 (cpu_class) and a6deff06 (env test); Mac clippy 7675bc8a; the full workspace suite on macOS in 4 shards 67564c5e, 275a4777, 34e74f87, 29d17494.
  • Red at 9f5eb21df89 (tests only): step 1f6e78b9. Both new tests fail on their assertion "the harness was darwinbg at its exec" (direct and hosted).
  • Green at f605bb4c7fb: step 56aaae87, all cpu_class tests pass. The existing cpu_class tests (background, normal opt-out, reference) run in the same filter.
  • agent_host tests on macOS at f605bb4c7fb: step 65ca15fe, pass.
  • Clippy on macOS at f605bb4c7fb: step db339509. Its only error is the known feat base red start_logged is never used in tests/daemon_memory.rs:82 (from #27991; fix #28418). This PR does not touch that file. The acpmux library, which holds the new macOS code, compiled and passed clippy.

Observed / Not run

  • A scratch probe of harness start time under the clamp (step d3d3a3a4) still runs; its numbers will be added here. They confirm the clamp timing but do not gate this PR.
  • Full macOS suite at 9a051797210 (shards 67564c5e, 275a4777, 34e74f87, 29d17494). Classification against the base 5a5aba66ae0 is pending (base shards b55ee72e, aae07cf6, dcf8974c, 985a7ca8; focused base/head pairs for the tests below).
    • Possibly this PR until the base runs answer: acpmux forum::a_rule_steered_into_a_running_turn_counts_as_told (timeout; base 3cd4b5ae, head 3c2d9752), acpmux agent_host::an_answer_does_not_settle_a_request_its_host_still_writes (timeout; its spec uses the Normal class; base 744369c4, head 90e8b523), acpmux hub_integration handoff::initialize_advertises_handoff_and_summaries_carry_enforcement (20 s timeout on initialize; base 6402bd38, head cc85e3df).
    • cmux-tui::setup harness_auth::cmux_setup_harness_auth_claude_check_failure_retains_profile_login_and_respects_disabled_launches: its profile checks with /bin/false, which macOS does not have (the check runs through a plain std Command, which this PR does not touch); expected a feat base red from #28187, confirming with base 21fa0ff8.
    • Unexplained, passed on retry (shard 3, job 34e74f87): stop::a_cancel_ends_an_honouring_agents_turn_within_the_budget, subsessions_safety::a_message_never_reaches_a_session_that_asks_less_than_its_sender, local_resize_order::local_resize_bounds_the_drain_to_a_queued_snapshot. Listed in followups.
    • Known base reds: hermetic_daemon_guard, noun_first_path, integrity_check_scope, agents_chat_tools::*, Mac clippy start_logged.
  • Not run: the Austin's-day A/B. It runs after review.

R9: no cap or deadline is added or changed. R-L5: the host start already ends on a signal (#28412).
perf claim: none

Performance

No perf numbers: the spawn timings above come from a scratch probe, not from an A/B of this head. The Austin's-day A/B runs after the review.

Changelog

Fixed: On a busy Mac, a new agent chat no longer waits for its agent process to start at the lowest priority; agents still run below the app's CPU class.

Proof

No capture: the change is in the acpmux daemon and has no UI.