acpmux: one update of a host at a time, through a lease every step checks

Merged
TE
14d ago
+809-37

Summary

Leo: "BRO CANT YOU JUST LIKE SUSPEND SHIT AFTER A TOOL CALL/TURN THEN JUST GO AND AUTOUPDATE OUR TEAMMATES NEED ACPMUX UPDATING TOO..."

This PR makes it safe for more than one thing to update a host's acpmux. That matters because #24070 (the app updating its ssh peers) and #24232 (the Linux self-update timer) can both pick the same host. It was split out of #24232, which keeps the timer and lands after this one.

  • One update per host at a time. host update, host setup and host enroll --fix take the host's update lock before they change its binary, acpmux.prev or daemon (host_lock.rs).
    • The lock is a lease in ~/.acpmux. update.owner names the update holding it (a random token). The process keeping it holds a flock on update.lease for its whole life, so the lease is free once that process ends, however it ends. That process is a script running over the update's own ssh connection, so if the connection drops, the lease is freed. A pid left in a file can't keep it.
    • Every script the update runs is fenced. Under a flock on update.lock, it first checks the lease is still its own, and otherwise runs nothing.
    • Uploads go to a name of the update's own and are moved in place by a fenced step. A late upload from an update that lost its lease never lands on what the next update staged.
    • The lock needs flock or perl on the host. With neither, the update stops with a clear error and changes nothing.
    • An update whose connection dropped stops with lockLost instead of racing the next update. That includes losing it during the restart or the health check, and then no rollback runs.
    • A host update that finds the lease held changes nothing and reports updateRunning. Setup and enroll --fix fail with the same message.
    • Before this, two overlapping updates could overwrite the known-good acpmux.prev with a bad build.
  • The daemon's own home and socket. Every update script reads ACPMUX_HOME and ACPMUX_SOCKET from systemctl --user show acpmux.service -p Environment --value. Values are split the way sh splits words, because systemd quotes values that contain spaces. An update now reaches a daemon that host migrate moved behind 10-migrated.conf (#24079). Before, it checked the default socket, judged the healthy daemon dead and rolled it back.

Leo-ask: ask-48bbaec1d4
Leo-ask: ask-061e3574a6
Leo-ask: ask-98502f19a4

Changelog

  • Fixed: two acpmux host updates of the same host no longer run at once. The second reports updateRunning and changes nothing, so the rollback binary stays the last good one. An update also checks a daemon that host migrate moved at the socket it really listens on.

Tests

  • host_lock_tests.rs (new):
    • two_updates_of_one_host_never_run_at_once;
    • an_update_whose_lock_connection_drops_changes_nothing_more;
    • of_updates_that_start_at_once_one_takes_the_lock;
    • a_late_copy_of_an_update_whose_lease_went_never_lands_on_the_next_ones;
    • a_stale_lease_whose_pid_now_names_another_process_is_free;
    • without_flock_perl_locks_and_without_either_nothing_runs;
    • the_daemons_settings_are_read_as_systemd_quotes_them;
    • a_daemon_whose_quoted_socket_has_a_space_is_checked_where_it_listens.
  • host_update_tests.rs: a_moved_daemon_is_reached_through_the_settings_its_move_carried. The fake peer now runs the lease connection, and the fake systemctl answers show.
  • host_enroll_tests.rs: enroll never relinks while another update runs, and the link step holds the lock.

Observed / Not run

  • Observed at eff3d4ff on cmux16: cargo test -p acpmux --lib cli::host passed 87 of 87 in each of three runs, and cargo clippy -p acpmux --lib --tests -- -D warnings printed no warning or error.
  • Observed at eff3d4ff: rustfmt --check and the prepush gate are clean. Staging cleanup uses find, not a shell glob, so a zsh login shell with no match no longer errors. The real lease scripts were run by hand in temp homes with flock(1) on Linux and with perl on macOS: a second holder is refused, another token is refused, the fence fails once the lease process is killed, and a stale owner naming a live pid is taken over.
  • CI at eff3d4ff: 104 passed, 4 red, all also red on base (Guard status, cmux-next checks roll-up, release-notary, risk evidence waiting on the review receipt).
  • Not run: no fleet host was updated by hand while developing, and no live daemon was restarted.
  • Size: 846 changed lines (+809/−37) in 9 files, about 420 of them tests. Just over the 800 target; the lock and its tests don't split cleanly.

No capture: no UI.

Agent session: Air-Blue

🤖 Generated with Claude Code