cmux NEXT: update acpmux ssh peers and Cloud machines to the app's build, also mid-turn

Closed
TE
10d ago
+7005-218

Status at fc47a99a: Codex (gpt xhigh) passed a8279f9b and confirmed the base merge fc47a99a only resolves the Debug Settings tunable list. Merges after #23199, then this branch takes base again.

Summary

Leo: "BRO CANT YOU JUST LIKE SUSPEND SHIT AFTER A TOOL CALL/TURN THEN JUST GO AND AUTOUPDATE OUR TEAMMATES NEED ACPMUX UPDATING TOO..."

Leo: "OUR SHIT IS HARDLY EVER GOING TO GO IDLE. IT JUST WONT. YOU'LL JUST HAVE TO ACCEPT IT BREAKING AT SOME POINT BUT IT RESUMES WHATEVER IT INTERRUPTED. END OF STORY FOR THE DAEMON UPDATE."

Stacked on #23199: only the commits after 74759d6f are this PR. Once #23199 merges, this branch takes feat-cmux-next again (a merge, not a rebase).

After an app update, the daemons on other machines move to the app's own build in the background, through one queue (DaemonUpdatePolicy.shared), one machine at a time:

  • acpmux ssh peers (huge-blue, cmux16, a teammate's box): acpmux --json host update NAME --binary-dir DIR on the app's bundled CLI. A peer on another platform gets the published cmux-tui-acpmux-<target> of the bundled build, downloaded and staged on this Mac. The CLI keeps the old binary and rolls back a daemon that does not come up with the expected build; it also moves a peer under a hand-written unit (huge-blue's acpmux-agent-huge-blue.service) to the standard acpmux.service (#23199).
  • Cloud machines: cmux link machine update --to-bundled, the same command as Update Machine. A Cloud update always installs the app's pinned build, never a newer or older one. The CLI swaps ~/.cmux/bin/cmux-tui, restarts only the daemon, and rolls back when the new daemon does not serve. A machine the app refused as incompatible goes first. machine_update_unsupported and machine_update_unavailable get their own messages.
  • No idle wait. A machine is updated mid-turn on the first ask. The restart keeps hosted agents (Linux units use KillMode=process, macOS agent hosts are in their own session), and the new daemon adopts them; a chat whose agent ended resumes from its saved session on its next prompt. A non-zero updateBusyAfter (tests only) brings back the idle-first path.
  • Peers that connect later are found. A peer that connects after launch is offered when its session appears, and a ten-minute sweep (AcpmuxPeerReconnects.sweep, while agent.updatePeers is on) offers every connected ssh peer again, so a peer with no session, or one whose offer was dropped while its earlier attempt still ran, is updated within ten minutes. The queue drops ids already queued or running, and the ledger keeps one update per build, so a sweep never updates a peer twice. The sweep ends when the toggle goes off or its sleep is cancelled.
  • A machine that cannot update now (unreachable, another update running, a peer the CLI cannot reach) is asked again with backoff: 2 minutes, doubling, at most 30, and left for the next launch after 6 hours. Its tooltip then drops "Update will try again soon".
  • A final outcome (updated, current, rolled back, failed) is kept per machine and build in a ledger, so a rollback or failure is not retried until the next build, and notifies once.
  • The machine's sidebar tooltip shows its state: updating, will try again, updated, failed, rolled back.
  • Debug Settings: agent.updatePeers (all machines) and cloud.updateMachines (Cloud only), both on, read before every step, so turning one off stops the rest.

Leo-ask: ask-48bbaec1d4
Leo-ask: ask-061e3574a6

Changelog

  • Changed: after cmux NEXT updates, the agent daemon on connected ssh and Cloud machines moves to the app's build in the background, also mid-turn, and an ssh machine gets its old build back if the new one fails. (Highlight: release-notes/next/highlights/connected-machines-update-with-the-app.md.)

Tests

  • DaemonUpdatePolicyTests: one at a time; urgent first; a busy target updated mid-turn on the first ask; a target that cannot update keeps the backoff (2, 4, 8, 16, then 11 × 30 minutes, 16 asks) and is left after six hours, unrecorded, with the board cleared; the ledger keeps final outcomes; one notice per failure; the toggle stops the queue. The withABusyCap… tests cover the idle-first path with a 600 s cap.
  • AcpmuxPeerUpdateTargetTests: the CLI arguments (no --if-idle while busy), reply parsing, needsBinary for another platform, and theFirstHostUpdateDropsIfIdle; through AcpmuxPeerReconnects, a peer that connects later in the launch is updated, and a peer that connects with no session is updated by the sweep, which ends when the toggle turns off.
  • CloudMachineUpdateTargetTests: a busy machine is updated at once with no quiet read; CLI error codes map to outcomes; an incompatible machine updates right away; a reconnect after every update.

Observed / Not run

Not run locally: no builds on this Mac, and cmux16 had too little disk for a package build. PR CI's compile and package-test jobs are the test run for this head.

Fixes after review (each re-reviewed by Codex): the toggle going off empties the queue before it reports (8b2694bc); a target offered while the toggle-off is reported still updates (a0534038); a peer the CLI cannot reach is asked again, not recorded as failed (08c033e8); a peer that connects later is updated (d00d7c36); a refused Cloud daemon on an app-server link is updated (a187752d); a Cloud header keeps a failed update while the machine is not on the app's build (e6ff99d0); the ten-minute sweep (1e97f0aa, a8279f9b). fc47a99a takes feat-cmux-next and keeps both base's HangFileTunables and this PR's CloudTunables in the Debug Settings catalog.

Run: python3 scripts/cmux-next/prepush-gate.py (no new violations), ./scripts/localize-changes (strict validator passed on the new retrying key), the What's New digest check on the highlight (no problems).

Not done: no live update of any real peer from this branch. Rolling the build out to the five hosts is acpmux-unify's part.

No capture: the only UI is a tooltip line on a machine's sidebar row and the Debug Settings toggles.

Agent session: Air-Blue

🤖 Generated with Claude Code