acpmux: an idle harness ends only when nothing but its MCP servers runs under it

Merged
TE
9d ago
+920-28

An idle harness used to be ended with Stop's group and tree cleanup, so a dev server or watcher it never reported died with it. Now the idle pass first lists what runs under the harness. It ends the harness only when every process there is one of the session's MCP servers or runs under one. Otherwise the harness stays up, and the pass logs why.

Leo, verbatim:

An idle Claude/Codex session (no turn running, nothing queued) hibernates after 2 minutes. Its process and its MCP sidecars exit, its state is checkpointed, and it wakes on the next prompt or click within about 1 s. Sessions waiting on a person stay cheap too: they keep only the question.

resource efficiency takes precedence over random first message latency.

Design B2, chosen after review round 8. Rounds 1-8 tried a PID-only retirement that spared the group. It needed a persisted record of what it spared plus drains on every end path, and round 8 found that structurally unfixable. B2 drops all of it: the hibernate frame, idle_pid_only, the absolute drain, the spared record and its drains. Ending the harness is base's detach_child again, guarded by a census.

What changed

  • Census (agent_host/tree.rs under): ps -axww -o pid=,ppid=,pgid=,stat=,args=. It takes every process under the harness by parent chain, and every process in the harness's process group, including one already reparented to init. Zombies are skipped. If ps fails, prints a line it can't read, or doesn't list a live leader, the census returns nothing and retirement is refused (fail closed). The daemon's own group is never taken as the harness's.

  • MCP sidecar set (hub/idle_census.rs): the servers the harness's configs named when it started. Just before each direct, hosted or pooled start spawns the harness, LaunchServers::read reads that launch's configs once (argv, folder, HOME, PATH, CLAUDE_CONFIG_DIR, CODEX_HOME), bounded by 3 s, and the start awaits it. A config edited after the harness starts never changes the set. If the read timed out, the harness stays up. The configs:

    • for Claude Code, the --mcp-config files in its argv (acpmux's own among them; a relative path is resolved against the harness's folder), plus inline JSON. Unless --strict-mcp-config, also .claude.json in $CLAUDE_CONFIG_DIR (else $HOME): the user mcpServers and the projects[<cwd>] entry. And the folder's .mcp.json;
    • for Codex, config.toml [mcp_servers] in $CODEX_HOME (else $HOME/.codex);
    • plus the mcpServers acpmux gave an ACP harness at session/new or load (ChildAgent::given_servers).

    A source that is missing or doesn't parse adds nothing, so a server only it names keeps the harness up. Only a direct child of the harness can be a server, and only when its command line is exactly the server's. Its program must be the configured command: the same path, or for a name, the first executable the harness's PATH finds (or the bare name). Alternatively a listed interpreter (node, bun, deno, python, ruby, perl, sh, bash, zsh, dash) runs that program as a script. Then come exactly the server's args. A bare interpreter configured with no args never matches. Neither does a server whose command or an arg is empty, holds whitespace, or isn't a string: ps prints argv joined by spaces, so the census couldn't tell such a server apart. Anything under a server counts too (npx, then node).

  • A harness another daemon started (adopted after a daemon update or restart) has no launch record. Its servers are unknown, so it ends idle only when nothing at all runs under it. With sidecars or any other child it stays up, which is safe. A census refusal only keeps the idle pass from ending a harness. Stop, close, shutdown and every other end path never consult it and are unchanged from base. Test durable_agents::idle_adopted::an_adopted_harness_with_a_child_stays_up_idle_and_a_stop_still_ends_both: after a restart, the adopted harness with a child stays up through several idle periods, and _acpmux/kill then ends both the harness and its child.

  • Bounded: the configs are read before the leases are taken, and both the reads and ps get a 3 s budget (CENSUS_BUDGET). Past it the harness stays up, so a stop never waits on a stuck read.

  • Refusal log: one info line with the blocking pid and command (or "could not be listed"), at most every 10 minutes per session (Session::idle_refusal_logged, a timestamp, not a timer).

  • Race fixes kept from rounds 1-5: the pass holds turn_lock, the new harness_use (write) and spawn_lock from its checks until the harness is ended. A start (ensure_child), a mode change (set_mode) and a session/fork through the parent's agent touch the session and hold harness_use (read). A stop (Hub::kill) holds it too, so it waits out an idle pass under way. The deadline and eligibility are checked again after the census. Eligibility also requires no harness-own turn (harness_turn), no ask in flight and no open secret card. stop_idle_reaper wakes the task and it exits once stopping. The pass only ever try_writes harness_use, so it never queues as a writer, and a nested read (a mode change that starts the agent) never waits behind it.

End-path table

"Leader" is the harness process (ChildAgent.pid, the harness pid for a hosted agent too).

#PathWhat happensCodeTest
1Idle pass, census clearbase detach_child: direct: group TERM, 300 ms, KILL, then the tree kill of what was under the unreaped leader; hosted: Terminate frame; lost link: end_unadopted_host. The MCP servers end with the grouphub/idle.rs stop_idle_childrenhub an_idle_harness_ends_only_once_nothing_but_its_mcp_servers_runs_under_it (the server is gone after the end, and the next prompt resumes)
2Census finds a process that is not a server (same group, setsid child, group member reparented to init, a server's command line under a shell)refused; touch; log linehub/idle_census.rs classify, runs; tree.rs under_inhub test (a same-group worker, then a setsid child, each keep it up); unit the_census_takes_children_group_members_and_skips_zombies, mcp_servers_and_what_runs_under_them_let_a_harness_end, the_configs_are_the_ones_the_harness_was_started_with
3Census not whole (ps fails or times out, unreadable line, output not UTF-8, leader missing or a zombie, pid unknown, config reads time out, Windows)refused; touch; log linetree.rs under; idle_census.rs idle_census, sidecarsunit a_census_that_is_not_whole_says_nothing
3aAdopted harness (no launch record)no servers known: ends only with nothing under it; otherwise refused, and every other end path is base'sidle_census.rs sidecarsdurable an_adopted_harness_with_a_child_stays_up_idle_and_a_stop_still_ends_both
4Session in use (attached, turn, harness turn, queued, permission, ask, secret card, background process or subagent, a lease held)refused; touchhub/idle.rshub an_idle_pass_never_ends_a_harness_under_a_mode_change; base background-task and lifecycle tests
5Activity during the censusthe deadline moved: not ended this passhub/idle.rs (re-check)none of its own
6Stop, close, purge, shutdown, MDM full-access end, route switch, failoverunchanged basebase tests
7Harness exits on its ownunchanged base exit watchers (group kill)base tests
8Daemon restart, crash or adoption mid-retirementmoot: B2 keeps no retirement state. A host adopted while its Terminate is under way is base's adopt-during-Terminatebase tests

The one remaining residual: the census is a snapshot. Between ps returning and detach_child's signal, with every lease held, the harness can change what runs under it while it reports idle: fork a process, or have one join its group or exec into something else. That process ends with the group. The window is one ps plus one detach_child.

A smaller named limit: the configs are read just before the spawn, so an edit in the moment between our read and the harness's own read is not seen. Only a server we read that the harness then dropped could matter, and only if a direct child of the harness runs its exact argv.

What this PR leaves as base does, all unchanged:

  • Work that escaped by double fork and setsid (ppid 1, its own group): neither the census nor the kill reaches it (both follow the same parent chain and group), so it keeps running.
  • A client attach after the final check and before the signal: the harness still ends, and the next prompt resumes it, as base's idle exit did.
  • ChildAgent::kill's group signal after the leader is reaped, and a crash mid-kill: Stop's existing path. B2 keeps no retirement state, so there is nothing to lose.

Known limits, all in the direction of keeping the harness up (each is logged):

  • an MCP server whose command line isn't exactly its configured command and args, such as a wrapper that re-execs under another name, or one an adopted harness runs;
  • servers that only Claude plugins name;
  • a harness that runs another process between itself and its servers (an ACP adapter running the claude CLI).

Windows has no census, so idle harnesses there stay up.

Timers and cleanup: one config read per harness start (LaunchServers::read, awaited before the spawn) in spawn_blocking under a 3 s tokio::time::timeout, which ends with the read; nothing outlives it. No new observer. The existing hub idle task runs one census per idle deadline per live child: one config read and one ps, each in spawn_blocking under a 3 s tokio::time::timeout. The timeout ends with the read or the ps, and when it fires the pass drops the result and moves on. Shutdown now wakes the task and it exits. The log rate limit is a stored timestamp.

Size: about 950 changed lines (920 added, 28 removed), over the ~800 cap. About 200 lines are the launch record (Launch, LaunchServers, read before the spawn at three start sites) and its tests, which rounds 11 and 12 required: without the configs as the harness read them at its start, the census can misjudge what is a server. Splitting it off would land a census known to misread configs.

Risk: normal. Idle harnesses end less often (only when the census is clear); ending is base's path. No visual change.

Performance

No perf numbers: the quoted latency is the existing trade-off of ending idle harnesses (a later prompt restarts one), which this PR leaves as base has it. What this PR adds is off the turn path: one config read per harness start (a few small files, bounded at 3 s) and, once per idle deadline per live child, one config merge and one ps, each bounded at 3 s and run in spawn_blocking. Nothing runs per token or per row.

Validation

  • Linux, at the exact head 22fa210 (run 40662460849): cargo fmt --check and cargo clippy -p acpmux --all-targets -D warnings are green. cargo test -p acpmux passes every new test: three census unit tests, three matching and config tests, hub idle_census x2 (including a config edited after the start), and durable idle_adopted. hub_integration passed 513 of 513 three times in the same run.
  • The reds that fail the same way on base 56b08cf (full base run 40654957218) are daemon_chat_open (I/O budget), ephemeral_run::a_probe_from_an_agents_shell_is_not_its_subagent, hermetic_daemon_guard x2, remote_guard_modes::a_web_source_is_resolved_like_its_handler_and_refused_when_it_cannot_be, and session_pool::pooled_sessions_end_with_the_daemon_and_after_a_crash (which passed at this head). Flaky on both: attach::upload::one_sender_at_a_time_holds_a_files_resume_record, codex_auth_broker, and the daemon_memory 160 MB budget (161 MB once). peer_integration::a_peer_chats_deny_goes_through_under_approve_reads_and_its_read_refuses_allows failed once under the full suite (the servedAs check right after add_peer). Alone it passed 6 of 6 at this head (run 40663942954) and 6 of 6 on base (run 40663945628). durable_agents::stalled_turn::an_adopted_turn_whose_model_went_quiet_ends_as_stalled_and_frees_its_agent failed twice under the full suite's load. It reads the status right after host_adopted, and base's adopt_one sets Ready before recover_work sets Running. Alone it passed 8 of 8 at this head (run 40656906084) and 8 of 8 on base (run 40656904967).
  • cmux-next fast gates: the no-process-scan ratchet allows one ps call in agent_host/tree.rs. The census shares it through one ps(fields) helper, and the baseline row's reason names the census. The local prepush gate over 56b08cf reports no new violations.
  • hub_integration::stalled_turn::a_turn_counts_from_its_prompt_being_written waited a fixed 300 ms for the idle exit. It now waits for the idle status, because the pass runs ps first.
  • Independent Codex gpt-6-astra review. Round 9 (xhigh, on the whole table) returned BLOCK at eec57c1. Fixed: the over-wide sidecar set (findings 1 and 2), lossy ps output (7), and the fork and Stop leases (5 and 9). Findings 3, 4, 6, 8 and 10 are the residual above or base paths this PR leaves unchanged. Round 10 (high, fast) returned BLOCK at 0b82d28 with three new defects. Fixed: the injected set is now what the harness was given; interpreters are checked and by-name commands resolve through PATH; the reads are bounded and done before the leases. Round 11 (high, fast) returned BLOCK at b743083 with five new matcher defects. Fixed as a class by the launch record, matching only direct children, exact argv, and a first-PATH-hit resolution. Round 12 (high, fast) returned BLOCK at ade25f2 with three new defects: configs re-read at idle time, args split on whitespace, and a unix-only import. Fixed: a snapshot at start, ambiguous servers never match, and a portable first_on_path. Round 13 (high, fast) returned BLOCK at fa39c83 with one new defect: the snapshot was read in a task after the start (hosted and pooled harnesses only after init), so an edit in between was trusted. Fixed: each start site awaits the read before it spawns the harness. Round 14 (high, fast): pending.

Agent session: agent session

šŸ¤– Generated with Claude Code