#25127 acpmux: a full disk or a crash loop never leaves the daemon down

Leo (2026-10-25), about huge-blue: "You know, this big freestyle VM, right? Like when we continue productionizing freestyle stuff, like say we also offer the persistent situation, say with CMux Cloud, like this is the sort of thing that we'll need to be, you know, just getting better and better at. Like both personally and as, like, users do it. Like we need to have good ass images, good ass ergonomics, good ass everything."

What broke

On huge-blue a runaway process tree filled the disk. The acpmux daemon then exited while it was loading:

Error: write ~/.acpmux-supervised/huge-blue/daemon.tmp-N-0: No space left on device

That write is the daemon.pid file (config::write_atomic names its temporary file <stem>.tmp-<pid>-<n>). It was the only state write on the start path that ended the daemon with ?. systemd restarted the daemon, it failed the same way, and after 5 failures in 60 s systemd stopped trying ("Start request repeated too quickly"). acpmux stayed down from 20:17 to 20:50 UTC, until someone restarted it by hand.

What changes

1. A failed state write no longer ends the daemon (daemon.rs, new daemon/persist_retry.rs)

• The daemon.pid write now goes through write_or_retry. A failure logs a warning, the daemon goes on serving, and the write is tried again after 1 s. The delay doubles up to a 60 s cap, until a write succeeds.
• The pid file only explains a dropped connection (client::closed_error), so serving without it costs nothing.
• Stop condition and cleanup of the new retry task: it stops at the first successful write. Shutdown aborts it and waits for it to end before it removes daemon.pid, so a late retry never writes the file back. While it waits at the 60 s cap it logs at most one warning a minute. An idle daemon on a healthy disk never starts the task, so it adds no wakeup (tests/daemon_idle.rs).
• On a multi-thread runtime the retry write runs in block_in_place, so a slow disk blocks only that worker. It runs inside the task, so a shutdown that aborts and joins the task never leaves a write running.
• config::write_atomic now removes its temporary file when the write, sync or rename fails. A full disk is no longer left holding a partial *.tmp-* file for each failed try.
• Other writes on the start path: I checked each one:
  • These already only warn: the generated web config and the moved listener (config.save()), the LocalApp and peer tokens, the login-env cache, and pairing (a read).
  • Writes after start (session meta, event appends) already warn and count the failure (hub/mod.rs).
  • Still fatal, unchanged by this PR: on a home that doesn't have them yet, creating the state folders can fail on a full disk. That covers sessions/ (from store::open), mailbox/ (from Mailbox::open, which otherwise only reads) and the empty daemon.lock. A home that has run once already has them. If one fails, the unit below now keeps retrying instead of giving up.

2. The host setup unit never gives up (cli/hosts.rs SYSTEMD_UNIT)

Before:

StartLimitIntervalSec=60
StartLimitBurst=5
Restart=on-failure
RestartSec=3

After:

StartLimitIntervalSec=0
Restart=always
RestartSec=3
RestartSteps=5
RestartMaxDelaySec=60

• The restart delay grows from 3 s to 60 s over 5 steps, and there is no start limit.
• RestartSteps= and RestartMaxDelaySec= need systemd 254 or newer. Ubuntu 24.04, which huge-blue runs, has 255. An older systemd logs Unknown key name 'RestartSteps' in section 'Service', ignoring. when it loads the unit, then restarts every 3 s with no limit. A warning is acceptable there; a unit that stays failed is not.
• Side effects of Restart=always, by design:
  • acpmux daemon shutdown on a unit host is followed by a restart. systemctl --user stop acpmux.service stops it for good.
  • A daemon that can't take the lock (another daemon holds the home) is retried at most once a minute, and takes over once that daemon is gone. Before, the unit gave up.
• KillMode=process stays: a restart still keeps the agent hosts.
• host setup now writes the unit beside its place and renames it in, so a full disk never leaves a half-written unit.
• The unit has no MemoryHigh. A test asserts that none is added, because MemoryHigh stalled the daemon for 37 minutes.

3. host update rewrites the unit on hosts set up earlier (unit_refresh, OLD_SYSTEMD_UNITS)

• #22899 writes the unit only at host setup. An update just restarted whatever unit the host already had, so without this step the new text would never reach a host that is already installed.
• Now, when ~/.config/systemd/user/acpmux.service matches the text #22899 wrote byte for byte, the update replaces it with the new text and runs daemon-reload before systemctl --user restart.
• The new text is written beside the unit and renamed over it, so a full disk never leaves a half-written unit.
• If daemon-reload fails, the old text goes back and the update goes on under the old unit; the next update tries again.
• A unit whose text is already current but that systemd hasn't read yet (NeedDaemonReload=yes) is reloaded before the restart. If that reload fails, or systemd can't be asked, the update stops before the restart (exit 1).
• If mktemp fails or the reference can't be written, the unit is treated as someone else's and left alone.
• A unit with any other text (someone edited it) is left alone, and so are drop-ins under acpmux.service.d/.
• The move from a hand-written unit (#22988, host_migrate.rs) treats a unit file an older acpmux wrote as acpmux's own (a rolled-back move leaves one), not as a conflict. It replaces that file the same way, written beside it and renamed over it.
• The move under a full disk:
  • Every file the move writes is checked: the settings it carries, the copied drop-ins, 10-migrated.conf and the planned unit.
  • A write that fails while planning refuses the move before anything changes (writeFailed).
  • Once the move has started, each file is written beside its place and renamed in (the runtime drop-in too).
  • A failed write undoes the whole step: the files this move created are removed, an earlier acpmux's unit text comes back, the runtime drop-in is removed, and so is the temporary folder. The script then exits 1 and the caller moves the daemon back.
  • The old text is kept beside the unit as .acpmux.service.acpmux-old, a name systemd ignores. It is written through a temporary name and renamed, on the same disk, so putting it back is a rename.
  • A failure before anything changes (mktemp, a planned file) refuses with writeFailed, so the dry probe before an update sees it and nothing is pushed.
  • A find that fails counts as a failed write; it never silently gives an empty list.
• Every Linux script (setup, update, refresh, migrate dry and real) is checked to parse under sh -n, dash -n and bash -n.

Evidence

The tests ran with cmux-ci test on a 32-vCPU Linux runner at the PR head (run 40234191959):

• cargo test -p acpmux --lib -- cli::host daemon:: config::: 132 passed.
• cargo test -p acpmux --test daemon_lifecycle: 5 passed.
• cargo clippy -p acpmux --all-targets -- -D warnings: clean.

What the new tests cover:

• tests/daemon_lifecycle.rs a_failed_state_write_while_loading_leaves_the_daemon_serving
  • It makes every daemon.pid write fail: that path is a folder, so the rename fails.
  • The real daemon binary still reports ready and answers _acpmux/status.
  • After 3.5 s of retries it is still up and answering, with no daemon.tmp-* file left behind, and it exits 0 on SIGTERM.
  • Before this PR the daemon exited with rename to …/daemon.pid before its ready line.
• daemon/persist_retry.rs (paused clock):
  • an ENOSPC writer returns control at once;
  • retries come at 1, 3, 7, 15 and 31 s and stop at the first success;
  • the delay caps at 60 s;
  • an aborted task writes nothing more.
• cli/hosts.rs:
  • assert_never_gives_up checks the unit text: StartLimitIntervalSec=0 sits in [Unit], Restart=always, the backoff keys are there, and there is no StartLimitBurst and no MemoryHigh.
  • linux_update_rewrites_a_unit_an_older_acpmux_wrote_and_keeps_anyone_elses covers the rewrite, then daemon-reload before the restart, and that an edited unit is kept.
• cli/host_update_tests.rs:
  • an_update_rewrites_the_unit_an_older_acpmux_wrote runs the full update against a fake peer, and drop-ins are kept;
  • a hand-written-unit move that finds an old acpmux-written acpmux.service adopts and replaces it;
  • with a cp that fails as on a full disk, whether copying the new text in or keeping the old text aside, the old unit stays whole and no temporary file is left;
  • when the disk fills at a drop-in after acpmux.service was already put in place, the move leaves no acpmux.service, no drop-in and no runtime drop-in.
• cli/hosts.rs failure paths: a failed reload, NeedDaemonReload=yes with the reload passing and with it failing, and no mktemp.

Risk

Normal.

• The daemon change only turns one fatal write into a logged one that is retried.
• The unit change applies on the next host setup or host update of a Linux host.
• Overlap: #24232 (self-update timer) also edits cli/hosts.rs, in other functions. Whichever PR lands second gets a base merge.

On huge-blue after this lands

1. Run acpmux host update huge-blue from a Mac on the new build, or host setup if its daemon is still under the hand-written acpmux-agent-huge-blue.service. The update moves it to acpmux.service (#22988), and that unit now has the new text.
2. If the daemon stays under the hand-written unit, that unit needs the same keys by hand:
   • StartLimitIntervalSec=0 under [Unit];
   • Restart=always, RestartSec=3, RestartSteps=5 and RestartMaxDelaySec=60 under [Service];
   • then systemctl --user daemon-reload.

Agent session: Air-Blue

🤖 Generated with Claude Code