PR #31593 - manaflow-ai/cmux

Summary

A NIGHTLY acpmux daemon on a fleet Mac held ~/.acpmux/daemon.lock for over two hours with ~/.acpmux/acpmux.sock gone. Every CLI call then failed with "holds daemon.lock but does not answer at acpmux.sock after 20s", and each new daemon run queued behind the lock. Nothing could reach the daemon, and nothing stopped it.

This PR fixes that stall (see Cause) and makes three socket changes:

Only the lock holder removes the socket. Before, server::bind_unix unlinked any socket that refused a connect. A live daemon whose accept queue is full refuses a connect the same way a dead socket does. Now bind_unix leaves the socket in place when another live process holds the lock that records it: this home's daemon.lock, or the one in the socket's folder, which covers an ACPMUX_SOCKET that names another home's socket (daemon::lock::socket_held_by_other). The probe tests the flock itself; the file only says who holds it and for which socket. The startup's error paths (the WebSocket bind fails after the unix bind) now remove the socket only while it is still the one this daemon bound.
A daemon whose socket alone is lost binds it again. The home watch now tells two losses apart (home_watch::Lost):
- Stop: the home, the lock or the socket's folder was lost. The daemon stops, as today.
- Socket: only the socket's own entry is gone, from the same folder, and the home and lock are still this daemon's.
On a Socket loss the daemon binds the same path again. The new listener takes the old one's place, and connections already open keep running. Then a new watch starts on the re-bound socket. The re-bind is bounded: it runs once per loss the watch sees (an event, never a timer), and at most 5 times per daemon life. Past that, or when the bind fails (for example another daemon now serves the path), the daemon stops as before. Each re-bind logs a warning that names the cause. A Socket loss no longer fences writes, because the home is still this daemon's.
The CLI says how to recover. "an acpmux daemon holds … but does not answer at … after 20s" now ends with the holder's pid and "To recover, stop it with kill -TERM PID: its agents outlive it and the next daemon adopts them". The start path's "still holds … and serves no socket" error says the same.

Cause

The daemon log and a sample of the stuck process (build 735d16bb5, 2026-11-01, which already has the macOS socket-folder watch) show the cause:

At 01:02:39Z the daemon began an ordinary, non-restart shutdown: stopping: the app of this daemon was gone for 3600s … end_agents=false (daemon/app_orphan.rs), then shutting down.
Before it stops agents, the shutdown removes its own socket (daemon.rs, "New clients get connection refused instead of a dying daemon"). So nobody else unlinked it, and the home watch had nothing to miss: the socket's removal was the daemon's own.
Next, Hub::shutdown_sessions ran stop_retry_at_shutdown for every chat. Each call is a synchronous backward scan of the chat's log (stop_auto_resume → LocalStore::scan_back). In the sample, the main thread, which runs the daemon future under block_on, sits in stop_auto_resume → scan_back (reads and huge allocations; peak footprint 2.1 GB). Nothing yielded, so the SHUTDOWN_BUDGET (5 s) timeout around that future could never fire.
The pass took 5 min 20 s (stopped at 01:07:59Z). For that whole time the daemon held daemon.lock with no socket. Every command waited out "holds daemon.lock but does not answer", and every new daemon run refused on the lock (… (pid 74646) / --ready-fd 12: Broken pipe) and exited. Then the daemon exited on its own, and the next start took over.

The ~10 threads blocked in flock in the sample are agent-host death watches (agent_host/wait.rs), which is normal. The three old daemon run processes still alive on that Mac started 2 to 11 days ago (the oldest on 2026-10-21, 12 MB, idle on a condvar). They predate the home watch and are not waiting on this home's lock. In current code a lock loser exits at once (the handover wait is bounded at 20 s, a unit's wait at 60 s per release).

So this PR fixes the stall itself and hardens the two socket paths the incident pointed at:

The shutdown's retry-stop pass yields, and ends at its own 1 s budget (hub/shutdown.rs). Each chat's log scan also checks that deadline on every line, so one large log cannot hold the shutdown either (the scan's skip passes the next line to visit, which ends the scan). Chats with a retry mark go first. Past the budget it logs how many chats it left; those keep only a retry their journal held that the startup had not restored yet. The daemon's 5 s shutdown deadline can now fire.
Every shutdown trigger logs stopping: <cause> (SIGTERM, SIGINT, a client's shutdown request; the app and unused watches already named theirs).

Testing

Red-first, all run with cmux-ci test on the Linux pool:

cargo test -p acpmux --lib shutdown_scan_tests: 20 chats whose backward scans each read 80 log lines at 50 ms a line (4 s a chat), then Hub::shutdown_all must return within 3 s. Red at 6ee14a1 (no budget at all) and at 4d58e2b (budget checked only between chats): the shutdown took 4.00…s. Green at the head.
cargo test -p acpmux --test daemon_lifecycle a_daemon_whose_socket_is_removed_binds_it_again: removes the socket under a live daemon, twice, in both layouts (the socket in the home, and in a folder of its own). Each time a client must reach the same daemon pid within 5 s. Red on base at 41033bc: sockgone: the daemon exited (ExitStatus(0)) after its socket was removed (round 1). Green at the head.
cargo test -p acpmux --test socket_owner: a live process holds a lock that records a socket refusing connects, and bind_unix from a process without that lock must leave the socket's inode in place. Red on base at 41033bc: a bind without the lock took the holder's socket path. Green at the head.

The old a_daemon_whose_socket_is_removed_stops is replaced by the re-bind test. The socket-ancestor retarget and home-loss tests still expect a stop, because a changed folder is a Stop loss.

Observed / Not run

Observed:

At the head 7133969: cargo test -p acpmux --lib -- shutdown home_watch daemon:: auto_resume retention (215 passed), --test daemon_lifecycle (18 passed), --test socket_owner (1 passed), and cargo clippy -p acpmux --all-targets -- -D warnings (clean). --test lost_home_writes passed at eaeb2c4 (5 passed).
The log tail and the process sample from the affected Mac (Cause above).

Not run:

The full acpmux suite.
A macOS run: the lifecycle tests ran on Linux (inotify). On a Mac the socket is seen through its folder's kqueue watch (cx-n1nrjk), which this PR leaves unchanged.
A re-bind on the affected Mac itself.

Changelog

Fixed: a stopping acpmux daemon no longer holds its lock for minutes with no socket. A daemon whose socket was removed binds it again, and a daemon of another home no longer removes a live daemon's socket.

Agent session: agent session

🤖 Generated with Claude Code