acpmux host update: carry ACPMUX_HOME when ~/.acpmux links to the unit's home

Merged
TE
15d ago
+70-14

deputy-2 reported, from the P0 rollout: "acpmux host update on huge-blue (about 300 sessions) reported "rolled back: the daemon did not answer within 20 s of its restart", because adopting that many sessions takes longer than the fixed 20 s health window. Make the window scale with session count, or make health answer before adoption finishes, with a test using a large fake session set."

The cause was something else. The update moves huge-blue's daemon from its hand-written acpmux-agent-huge-blue.service to acpmux.service. huge-blue's journal from that run (02:05:22 to 02:05:38 UTC, 2026-10-22) shows the new unit never got as far as adopting anything:

acpmux.service: Started acpmux.service - acpmux daemon.
acpmux[2079403]: Error: open private state directory /home/leo/.acpmux
acpmux[2079403]: Caused by: Not a directory (os error 20)
... (5 times, 3 s apart) ...
acpmux.service: Start request repeated too quickly.

On huge-blue, ~/.acpmux is a link to ~/.acpmux-supervised/huge-blue, and the old unit sets ACPMUX_HOME to that folder. The migration plan copies ACPMUX_HOME into 10-migrated.conf only when it is not the default, and it compared the two paths with readlink -f, so the link made them look equal. Without ACPMUX_HOME, acpmux.service opens ~/.acpmux, and the daemon refuses a home reached through a link (config::private_state_dir, O_NOFOLLOW). The unit crash-looped, the 20 s wait ran out, and the update rolled back.

The health window was not the problem. The daemon serves its unix socket, and so daemon status, before it loads chat logs or adopts hosts (daemon.rs, the spawned load_logs_async then adopt_agent_hosts). On the 01:57 restart, huge-blue's daemon logged ready 10 s before "startup complete". So the window stays as it is.

Change

  • cli/host_migrate.rs: compare ACPMUX_HOME byte for byte with $dh (REMOTE_HOME, ~/.acpmux), not through links, so huge-blue's home is carried to acpmux.service. A trailing / counts as different.
  • ACPMUX_SOCKET is now carried whenever the old unit sets it. The daemon treats an explicit socket and the default one differently: a path that is too long falls back to one derived from the variable, not from the home.
  • A drop-in of the old unit named 10-migrated.conf is refused (unitSettings) before anything changes, because it would overwrite the carried settings. The carried drop-in is written after the unit's own drop-ins are copied.
  • cli/host_update_tests.rs: the fake systemd now does what the daemon does. acpmux.service never answers when ~/.acpmux is a link and no drop-in sets ACPMUX_HOME. Restart and enable go through one start, which reports success the way a Type=simple unit does.
  • huge_blues_stale_binary_never_runs now expects the carried home, where it used to assert that nothing was carried, which was the bug. It also covers a home given with a trailing / and a clashing drop-in.

Red, then green

cargo test -p acpmux --lib -- cli::hosts

  • Red commit 7dc5cfbcb37 (test only), on base 9fe0c23896e: huge_blues_stale_binary_never_runs and an_acpmux_service_someone_else_wrote_is_refused_and_a_matching_one_adopted fail with outcome: RolledBack, reason: noAnswer, error: "the daemon did not answer within 20 s of its restart", the error huge-blue gave.
  • Head: 30 passed, 0 failed. Reverting the home comparison to ${h%/} turns the trailing-/ case red again.

Review

Codex at xhigh effort (codex exec, read-only), three rounds. Round 1 blocked on the trailing / and on the fake restart; round 2 blocked on the socket comparison and on a clashing 10-migrated.conf. All four are fixed. Round 3 gave SHIP at 96f10ef4420.

Observed / Not run

  • Observed: the focused hosts tests above on huge-blue, cargo fmt --check, scripts/cmux-next/prepush-gate.py (no new violations), and huge-blue's journal for the failed run.
  • Not run: a real host update huge-blue with this build. huge-blue's daemon was updated by hand to fcacd19 on 10-22 and still runs under acpmux-agent-huge-blue.service. The --if-idle part of deputy-2's report is a separate PR.

Changelog

Fixed: acpmux host update moves a daemon whose ~/.acpmux is a link to its unit's home without rolling back.

Agent session: huge-blue