cmux-next: local acpmux and cmux-tui move to the bundled build by themselves

Closed
TE
10d ago
+3419-200

Status at 05367c0d: Codex (gpt xhigh) passes this head. The acpmux watchdog sends its signal by the daemon's audit token (proc_signal_with_audittoken, looked up at run time because macOS 14.0 and 14.1 lack it), so a pid handed to another process is never signalled however long the app stalls.

Leo-ask: ask-98502f19a4
Leo-ask: ask-48bbaec1d4
Leo-ask: ask-e8a08a017c

Leo: "OUR SHIT IS HARDLY EVER GOING TO GO IDLE. IT JUST WONT. YOU'LL JUST HAVE TO ACCEPT IT BREAKING AT SOME POINT BUT IT RESUMES WHATEVER IT INTERRUPTED. END OF STORY FOR THE DAEMON UPDATE."

So no wait here is open-ended: a stale cmux-tui waits at most 2 minutes for a quiet moment and a stale acpmux with agent hosts at most 10 minutes, then each is handed off between requests. #24026 then restarts a stale local acpmux at once and shows its build in Settings > Updates.

Summary

After a cmux NEXT update (nightly, beta or release), the app's own daemons could stay on the old build. A local acpmux with no agent hosts was kept running indefinitely ("its agents would end"). A cmux-tui found stale after launch waited for the manual Restart button. A daemon that was alive but hung had to be restarted by hand. This PR is the local-app half of keeping every daemon on the bundled build:

  • Stale acpmux. At every lookup (launch and each reconnect), a daemon of another build is handed off once no session is running a turn or waiting on a permission. This now includes daemons with no agent hosts; their chats resume from disk at the next prompt.
    • While it is busy, the app waits on a one-shot _acpmux/watch. A daemon with agent hosts waits at most 10 minutes; the wait races a deadline on that cap, so a long turn that only streams output can't hold it past the cap.
    • Health check: the next daemon must report the bundled build within 5 s. It gets one retry, then the "acpmux did not start" banner shows.
  • Hung acpmux. If 3 status calls miss their deadline across 20 s, a 5 s confirm runs, then SIGTERM. SIGKILL follows after a 10 s exit wait. This happens at most once per pid. Agent hosts survive SIGTERM and the next daemon adopts them.
    • The daemon is known by its pid and start time (proc_pidinfo), checked again right before each signal and only after the kernel exit watch is on the pid. A pid reused by another process counts as the daemon's exit and is never signalled.
  • Stale cmux-tui. QuietDaemonUpdate checks every handshake against the bundled build. The build is read again each time, so an in-place app update is noticed. A stale daemon is handed off with the existing lossless handOffForUpdate once quiet:
    • Quiet means the person has been idle 60 s or the app is in the background, no menu or drag is in progress, no app request is in flight, and the owner has no CLI, MCP, chief-chat or TUI client attached.
    • It waits for that for at most 2 minutes from when the daemon was found stale. After that, only an app request in flight or a menu, drag or resize loop defers it, retrying every 5 s. An attached TUI exits and reattaches; an attached cmux chief chat exits (code 3) and a CLI or MCP call in flight fails, and both can be run again.
    • Health check: the new daemon must report the bundled build within 90 s. It gets one retry, then a "Couldn't restart cmux-tui" toast.
  • Hung cmux-tui owner. After 2 or more stuck server ensure results over 30 s, the owner is replaced (--replace-unresponsive) with no confirm. This happens once per owner process and at most 3 times per launch. An owner answering server.busy is never replaced.

Each behaviour has a default-on Debug Settings toggle: agent.replaceStaleAcpmuxWhenQuiet, agent.restartUnresponsiveAcpmux, cmuxTUI.restartStaleWhenQuiet, cmuxTUI.restartUnresponsive. The rules are written up in plans/cmux-next/server-updates-DESIGN.md (rule 2a/2b and the status table).

Coming next, in separate PRs:

  • acpmux host update with rollback, idle check, arch check and migration of hand-written units;
  • automatic updates of ssh peers and cloud machines to the app's pinned build;
  • a daemon-capabilities file for web-ota compatibility.

Testing

Tests added:

  • CmuxNextAgentPaneTests
    • AcpmuxQuietHandoffTests: a stale daemon without agent hosts, busy then quiet, the agent-host cap, the health check with retry and banner; aBusyDaemonWithAgentHostsIsHandedOffAtTheCapWithNoEvent (the cap fires with no event and no reconnect, and not before).
    • AcpmuxUnresponsiveWatchTests: the miss window, confirm, once per pid, toggle off. AcpmuxUnresponsiveIdentityTests: a reused pid is never signalled; a daemon that exits after its identity check gets no signal (aDaemonThatExitsAfterItsCheckGetsNoSignal); a signal whose token names an earlier process with the pid is refused by the kernel (aSignalForAnotherProcessWithThePidIsRefused).
  • CmuxNextDaemonTests
    • UnresponsiveOwnerWatchTests, IdleOwnerRestartRetryTests.
    • QuietUpdateKeepsTerminalTests: a live cmux-tui in a temp session. The quiet restart waits for a CLI call and the terminal keeps running.
  • CmuxNextAppTests
    • QuietDaemonUpdateTests: covers an old daemon next to an app updated while running, a successor of the wrong build or one that never connects, anAttachedClientBetweenTurnsHoldsTheRestartOnlyUntilTheCap and pastTheCapARequestInFlightStillDefersIt.
    • DaemonAutoRestartTests, plus ServerUpdateNoticeTests.anotherBuildWithNothingMissingIsAPlainNotice.

Observed / Not run

  • CI's first test run found that the quiet cmux-tui restart could never fire. The daemon's own agent-screen detector holds two kindless connections forever, and the idle probe counted each kindless local client as a CLI call. Now a kindless client counts as a call only for its first 60 s (4f7d43fb).
  • On cmux16 (macOS 27), swift build --build-tests of the package passes at the new head:
    • QuietUpdateKeepsTerminalTests passed 3/3 and QuietDaemonUpdateTests 5/5 (at 4f7d43fb);
    • AcpmuxQuietHandoffTests, AcpmuxQuietHandoffDecisionTests, AcpmuxUnresponsiveWatchTests, AcpmuxUnresponsiveRestartTests and base's AcpmuxStrayDaemonTests passed 3/3.
  • Those runs found three fixture bugs in this PR's own tests (90860f53). CI never finished those suites because its test helper died first.
    • The stand-in daemon's ready line used a ws:// URL, which the launcher refuses.
    • macOS 27 kills a copied /usr/bin/nc, so the copy is now signed ad hoc.
    • The old daemon's exit event can come before the process table shows it gone, so that check now waits (bounded).
  • CI's UnresponsiveOwnerWatchTests, IdleOwnerRestartRetryTests, QuietUpdateKeepsTerminalTests and OwnerRestartWaitsForChatTests passed in the daemon-tests job.
  • At c38a3470 on cmux16: the named suites above passed, 49 tests in 8 suites. ClassicUpgradeSimulationTests was not run there: the build guard refused it for low disk, so CI's app-tests job is its run.
  • Not run outside CI: the watchdog commits after c38a3470 (f8f10fc5: the kqueue exit watch before the identity check; d969314f and 05367c0d: the audit-token signal). cmux16 had too little disk for another package build, so CI's package-test lane is their only run. aSignalForAnotherProcessWithThePidIsRefused expects the kernel's refusal, so it would fail on macOS 14.0 or 14.1, where the token call is missing and nothing is sent; no CI runner uses those versions.
  • Never run against a live daemon: no daemon on any of Leo's machines was restarted while developing this.

No capture: there's no visible UI to show. The change is daemon upkeep (handoffs, restarts and watchdogs) behind the Debug toggles.

Changelog

Changed: cmux NEXT moves its local acpmux and cmux-tui daemons to the app's own build after an update, at a quiet moment or after at most a few minutes, and restarts either one if it stops answering

Proof

No UI change beyond the existing toasts and banner; behaviour is covered by the tests above.

Checklist

  • Behavior changes have added or updated tests
  • Localization: the Debug toggle titles (one renamed to "Update a stale cmux-tui automatically") are debug-only. The cmux-tui failure reasons are English inside the localized "Couldn't restart %1$@: %2$@" (follow-up).
  • Reviewed with a subagent before merge

Agent session: Air-Blue

šŸ¤– Generated with Claude Code