acpmux: an idle agent host with no daemon exits after a grace

Merged
TE
6d ago
+796-28

Part of Leo's ask: "HOW THE F*** IS THIS STILL NOT FING CHANGED IT'S LITERALLY "STOP SHOWING FING NONSENSE ON LOAD" + "JUST FING REMOVE THE FING OLD DAMEMONS WHAT THE…""

What

acpmux agent hosts (__agent-host) outlive their daemon on purpose (an update, a crash, a restart; the app's Quit hands them to the next launch). One whose daemon never came back (a tagged build's home that is never relaunched, a daemon killed for good) ran forever.

A host now ends its harness group, the way Terminate does (SIGTERM, SIGKILL after 2 s), and then itself, removing its record, socket and lock as any exit does, only when all of these hold:

  • no controller has been connected for its wait. The wait is agent_host::ORPHAN_IDLE (10 min), or the daemon's own idle period when that is longer. That is longer than the daemon's idle hibernation (2 min, hub::IDLE_CHILD), so an idle agent ends no sooner than a running daemon would have hibernated it. A daemon restart or update reconnects within seconds, and its hello starts the wait again.
  • its agent is idle by everything the host can see: no request the controller sent waits for its answer (a turn in flight); no fenced line waits for its receipt; every entry a next daemon needs is acked (all but stderr lines, raw Tap in lines and the exit); and Claude Code runs no subagent or background shell and holds no carried output.
  • the idle pass's census (hub/idle_census.rs, the one #27681 hibernation uses) finds nothing under the harness but the harness itself and its MCP servers: the ones its configs name, plus the ACP mcpServers its sessions opened with. A dev server the agent left keeps the host up, as it keeps a running daemon from hibernating it.

The census answer then goes round the host's event queue once more, and the host checks everything again before it ends the harness, so a line or a connection that arrived meanwhile keeps the host. If the host can't tell any of these, it keeps running.

While the harness is ending:

  • A daemon that connects gets its hello answered only once the harness is reaped. So it never starts a turn on a dying harness. It logs the exit, and the host then ends as every host does.
  • Anything the harness writes as it ends that a next daemon needs is kept, with the exit, for one more wait, for a daemon to come and log it.
  • A connection that has not said hello yet gets 5 s (the controller's own hello budget).
  • The end has its own deadline (the SIGTERM grace plus one more wait). Output held back by the buffer cap can therefore never keep the host forever. A busy host looks again one wait later. The daemon passes the policy in a new SpawnSpec::orphan_exit field (#[serde(default)]). It passes none while idle hibernation is off (ACPMUX_IDLE_CHILD_SECS=0), so an agent the daemon would keep is kept by its host too. A daemon that predates this field never sets it.

Ending an idle host is the same outcome as #27681's hibernation. The next daemon finds no host and loads the chat idle (store_load.rs), and the next prompt resumes it from the harness's own transcript. No new noise on load.

Not changed: case (c), a daemon whose home directory is gone. Hosts already end for that: watch_record in agent_host/host.rs ends a host as soon as its record goes, and host_lifetime.rs a_kept_host_ends_when_its_home_is_deleted covers it.

Debug-only test seam: ACPMUX_TEST_HOST_ORPHAN_IDLE_MS (daemon side, hub/idle.rs orphan_exit) replaces the wait. The orphan logic is in a new agent_host/host/orphan.rs, so host.rs (already over the god-file budget on base) does not grow (1215 lines before and after).

Prior art: #27681 (daemon-side idle hibernation, merged). Not duplicated by #27801 (a DEV daemon whose app is gone stops and ends its agents) or #27796 (a daemon whose home is gone stops); this is the host side, for when no daemon runs at all.

Tests

Red, then green: cd cmux-tui && cargo test -p acpmux --no-fail-fast --test agent_host --test host_lifetime -- orphan no_daemon_came_back

  • At the test commit e40b87f5ce3 (plumbing and tests, no behavior), 3 failed and 1 passed. orphan_exit::an_idle_host_whose_controller_is_gone_ends_after_its_wait failed with "the idle host outlived its wait". orphan_exit::a_host_keeps_a_running_turn_and_its_unlogged_end_for_the_next_daemon failed with "the idle host outlived its wait". a_kept_idle_host_ends_once_no_daemon_came_back_for_its_wait (a real daemon, SIGKILLed) failed with "host ... outlived its wait". a_host_with_its_controller_connected_never_ends_for_idleness passed.
  • At the fix commit, all 4 pass.

Tests added after review, all passing at the head:

  • a_host_no_controller_ever_reached_ends_after_its_wait
  • a_daemon_connecting_while_the_harness_ends_is_told_it_exited
  • what_the_harness_writes_as_it_ends_waits_for_a_daemon
  • an_end_whose_output_the_buffer_cap_holds_back_still_ends

Clean-runner receipt at the head b5f268ba9f54: https://github.com/manaflow-ai/cmux/actions/runs/41074425439 (cmux-ci test: cargo test -p acpmux --no-fail-fast --test agent_host --test host_lifetime && cargo test -p acpmux --lib -- orphan census agent_host idle). It passed. All of agent_host and host_lifetime pass there, including the four that fail on the lane VM.

Unit tests (cargo test -p acpmux --lib -- orphan census agent_host idle): 31 passed. They include an_orphaned_host_ends_only_an_idle_agent_after_its_wait and only_what_a_next_daemon_needs_keeps_an_orphaned_host (the decision function and the entry filter). cargo clippy -p acpmux --all-targets -- -D warnings is clean.

Observed / Not run

  • Observed on a Linux lane VM: the commands above, plus all of --test agent_host --test host_lifetime --test durable_agents. Five tests fail there and fail the same way with the fix's sources reverted: a_start_waits_for_a_harness_binary_an_update_is_replacing, a_kept_host_ends_when_its_home_is_deleted, a_pooled_host_ends_with_its_daemon_and_a_taken_one_stays, a_pooled_host_is_idle_and_follows_its_record, stalled_turn::an_adopted_turn_whose_model_went_quiet_ends_as_stalled_and_frees_its_agent. That VM's user had 125 of 128 inotify instances in use, so host record watches could not start.
  • Not run: macOS. The logic is Unix-generic (tree::under uses ps), but no macOS run was done.
  • Review: an independent codex gpt-6.1-sol review went seven rounds and ends with SHIP at the head (comment below).
  • Known limit: a host whose controller sent a request the harness never answers stays up. One case is a codex-acp steer, where only the last steered prompt gets an answer. That is the safe failure ("if it can't tell, it keeps running").

Risk: normal (acpmux host lifecycle; an idle agent ends only after its wait, and nothing ends mid-turn).

Changelog

Fixed: an idle agent left by an acpmux daemon that never restarted no longer runs forever; it ends after 10 minutes and resumes from its transcript on the next prompt.

Agent session: agent session

🤖 Generated with Claude Code