acpmux: a named new --reuse reconnects when the daemon restarts during its create

Merged
TE
1d ago
+395-82

Leo, 2026-11-05: "I thought we had better typed errors than this. I thought we had a better solution than whatever was going on here."

A NEXT Cloud chat starts its agent through the machine relay, which runs acp --json new -d --reuse -m claude -n relay-<chat> --cwd .... When the user's acpmux daemon restarts while that session/new waits (the app's keyed acpmux start stops acpmux.service and shuts the daemon down with --keep-agents), the connection closes with no reply. acp new failed with the retryable daemon_closed error, the relay sent chat.failed, and the chat was dead. Production saw it after 102.7 s (cmuxterm-hq run 42112686310). On staging, cmuxterm-hq cloud-platform-probe run 42117783639 repros the trigger (a restart during session/new), and run 42117848092, a SIGTERM, gives production's text word for word ("the daemon stopped and left its socket behind"). The next chat on the same machine failed the same way 3 s later: the systemd-restarted daemon took over a second daemon right as the next acp new connected to it.

Now a named new --reuse reconnects and runs its whole find-or-create again: it first looks the name up, so a session the first try already created is found and no second one is made. It does this up to twice, so two handovers in a row (the staging case) are covered too. A new without --reuse and a name does not retry, since nothing could find a session the lost create made. ensure NAME, which is the same find-or-create, now reconnects the same way. Prompts already reconnect like this (queue_prompt, wait).

This does not touch cmux-host's relay (#36828 changes it).

Changelog

  • Fixed: a Cloud chat no longer fails when its machine's agent service restarts while the chat starts.

Tests

cmux-ci test -- cargo test -p acpmux --test new_across_restart (new file, next to send_across_restart.rs). A fake daemon on the CLI's socket closes the connection with no reply on session/new, as a restarted daemon does:

  • the create happened before the close: the launch succeeds, prints that session, and the daemon got one session/new and holds one session named relay-x;
  • the close came before the create: the retry makes it;
  • two closes in a row, the second after its create: still one session;
  • without --reuse: no retry, the launch fails with the close, one session/new;
  • ensure, closed after its create: finds the session, one session/new.

The test commit comes before the fix. In one fleet run, cmux-ci test at head 16dd49e (https://github.com/manaflow-ai/cmux/actions/runs/42123932720):

  • with the fix reverted (the test commit b1eff657f57's sources): 4 failed, 1 passed (a_launch_without_reuse_does_not_retry_its_create, which passes both ways and guards the no-retry path). Each failure is production's error word for word: {"code":"runtime","detail":"daemon_closed",...,"message":"daemon connection closed while waiting for the reply to session/new: the daemon stopped and left its socket behind. ..."};
  • at the head: 5 passed.

cargo test -p acpmux --test send_across_restart at the head: 30 passed (run 42119897474). An earlier red step in that run passed by mistake: the CI snapshot commit's parent is the head, so HEAD~1 there was the fix itself, not the test commit.

Observed / Not run

Observed: the runs above. Not run: the staging repro itself against this build; nothing was built on the lane VM.

Agent session: agent session

šŸ¤– Generated with Claude Code