#27594 acpmux: a daemon whose home is gone stops; autostart never recreates a removed home

Leo: "Yes, fix it" / "Yes, do both".

Root cause

Two acpmux daemon run --ready-fd 12 processes from the NIGHTLY bundle ran for 7-8 days on one Mac with ACPMUX_HOME under a removed Python mkdtemp folder (.../T/tmp7uxt3tq_/.acpmux).

1. A test autostarted the first daemon under HOME=<tmpdir>, then removed the temporary home. The daemon kept running on deleted files. It stops only on SIGTERM/SIGINT, the shutdown RPC or a listener failure, so it never stopped.
2. Later a CLI call with ACPMUX_HOME/ACPMUX_SOCKET pointing at the removed path autostarted a second one. spawn_detached re-created the folder (private_state_dir), the new daemon.lock was a new inode and so was free, and a second daemon started next to the first.
3. A related bug: socket_path() honours ACPMUX_SOCKET even when ACPMUX_HOME is unset. A command with only the Chief's socket took ~/.acpmux/daemon.lock for that socket. It then waited the whole 20 s handover budget on a lock held by a daemon that serves another socket, and the error named ~/.acpmux/daemon.lock together with ~/.cmux/chief/default/acpmux/acpmux.sock.

Fix

• A daemon whose home is gone stops (daemon/home_watch.rs). When it starts, the daemon records the device and inode of its home, its socket and its daemon.lock. It shuts down normally (no restart handoff) once any of the three is removed, renamed away or replaced.
  • The old paths may already belong to another daemon, so the shutdown is careful with them:
    • config::mark_home_lost is set the moment the loss is seen. After that, private_state_dir/_rw/_append, write_atomic, owner_only::create_dir_all, the registry cache and store segment rotation refuse to write under the old home, so the last saves never re-create a removed home or write into its replacement.
    • Agents are ended, since no later daemon can adopt them from there.
    • Only the socket this daemon bound is removed (identity checked). The pid and auth files are left alone.
    • The router is let go without asking the path-based admin socket to stop.
  • Known limit (noted in the module docs): renaming an ancestor of the home changes nothing in the watched folders, so it is only noticed at the next change there. Watching every ancestor would wake the daemon for activity above it, and FSEvents watches a folder's whole tree. Autostart also refuses the old path once it is missing.
• Autostart never re-creates a missing non-default home (config::autostart_refusal). A command that would autostart (ls, send, an MCP server, ...) refuses a missing ACPMUX_HOME unless it is ~/.acpmux or the build's own home (set_home_override). An explicit acpmux daemon start, or a keyed start, still creates its home.
• Lock and socket come from one home.
  • ACPMUX_SOCKET pairs only with an explicit ACPMUX_HOME, or must be the home's own socket. This is the app's rule (AcpmuxEnvironment.swift).
  • A command with only a foreign socket now fails at once, naming both sockets, instead of starting a daemon under the wrong lock.
  • The daemon now hands ACPMUX_HOME out next to every ACPMUX_SOCKET it gives agents and MCP servers (agent/command.rs, agent_tools.rs), so they keep autostarting the right home.
  • daemon.lock now records the socket its holder serves on line 2. The pid is still on line 1, which is the only line holder_pid reads. When the home's daemon serves a different socket (its recorded one, or the home's default), a command now fails before any start, naming that socket, instead of waiting out the 20 s handover.
  • No Chief code is touched: optchat-chief already sets both variables.
• The daemon's self-update path is unchanged.

Timers and watchers

What	Wakes on	Stop condition	Cleanup
home_watch::watch: one notify watcher (inotify on Linux, FSEvents on macOS), NonRecursive, on the home and on the socket's folder when that is a different folder	entry changes in those folders only; each event re-reads three lstats and nothing else	the daemon's shutdown select! ends (any stop), or the watch reports a loss	the watcher is owned by the HomeGone future, which is dropped when the shutdown select! ends, before the shutdown work; dropping it stops notify's thread

No timer, no poll. An idle daemon's folders see no entry changes, so it adds no idle wakeup (tests/daemon_idle.rs). If the three paths cannot be read at start (a relative path), nothing is watched and nothing is ever counted as a loss.

Review

Independent review by Codex gpt-6-astra (high, fast) through review-router, three rounds. Round 1 found the lost-home shutdown touching a possible successor's files, the wrong-socket check running after the child's own handover wait, and the ancestor-rename gap. Round 2 found store segment rotation, the router stop, and custom-socket holders. All are addressed above; the ancestor rename is documented and was judged acceptable.

Tests

• tests/daemon_lifecycle.rs
  • a_daemon_whose_home_is_removed_stops: the home is removed and the daemon exits with success within daemon::EXIT_BUDGET. That budget is the existing SHUTDOWN_BUDGET + 8 s margin that wait_for_exit already used, now named.
  • a_daemon_whose_home_is_replaced_stops: the home is moved away and re-made; the daemon exits and does not bind the new home's socket.
  • a_command_in_a_missing_home_starts_no_daemon: ACPMUX_HOME=<missing> acpmux ls fails, says the home does not exist, and does not re-create it.
  • a_socket_without_its_home_starts_no_daemon: a lone ACPMUX_SOCKET fails at once, names the socket, and creates no daemon.lock.
  • a_socket_the_homes_daemon_does_not_serve_fails_fast: a wrong ACPMUX_SOCKET, and a holder on a custom socket, both fail within HANDOVER_BUDGET, naming the socket that is served.
  • The removed-home test runs with a disk store and a live chat, and checks the home is not re-created. The replaced-home test checks nothing is written into the new home.
  • Red/green: with the watch and the refusal turned off, the four new daemon/CLI tests fail. In that red run the two autostarted daemons kept running under the removed test homes, which is the original bug reproduced on gcp-agent-lane-2; they were stopped afterwards.
  • Also green on gcp-agent-lane-2 (umask 022): daemon_idle, daemon_start_race, daemon_single_owner, daemon_list_guard, daemon_restart_close, cli_ergonomics, session_env, inherit_start, send_across_restart, send_no_wait_old_daemon, start_defaults, subsessions, stats, agent_tools, harness_restart, session_lineage, and the acpmux lib (1935/1936; the one failure is linux_sandbox::proxy hitting that box's ulimit -n 1024, a file this PR does not touch). clippy --all-targets is clean.
• Unit tests:
  • daemon::home_watch::tests: a removed or replaced path counts as lost; a path that cannot be read at start is never watched.
  • config::autostart_pair_tests: the missing-home rule and the socket/home pairing rule.

Agent session: agent session

🤖 Generated with Claude Code