fix(next): no 'cmux-tui could not start again' notice while cmux is still restarting it

Merged
TE
2d ago
+289-8

Leo, on the notice "cmux-tui stopped and could not start again. cmux keeps trying. [Restart cmux-tui]" over a window of a NEXT nightly on a heavily loaded Mac: "why did this show up".

Why it showed

DaemonRecoveryNotice (#30496, restore fuzz h2) shows the moment DaemonRecovery.offersRestart is true while the local daemon is disconnected, and offersRestart turns true on the first failed server ensure. So a single reconnect attempt that failed (a missed heartbeat or a slow restart on a Mac at load ~400, a version handoff) put up a "could not start again" notice while cmux was still retrying by itself, and usually recovered moments later. The link machine update --to-bundled --if-older process that was running at the time updates a Cloud machine over its remote connection; it does not touch the local daemon.

The fix

  • EnsureOutcome counts the failed ensures in a row (failedInARow) and tells its observer after every failure, not only when the failure text changes.

  • DaemonRecovery.startFailurePersists: a full disk counts as down at once (only the person can free space). Any other failure counts only after 3 failed starts over at least 2 minutes (NoticePolicy.standard). The local reconnect tries at least every 30 s, so a start that really stays broken is reported within about 2.5 minutes.

  • The notice over content shows only when startFailurePersists is true, or after the person's own Restart failed. Restart stays offered meanwhile (the launch window's controls and debug.daemon are unchanged; debug.daemon adds start_failure_persists).

  • The wording stays as is: once shown, it is accurate (what is wrong, and the one action), and it is already translated.

  • Logging: every variant (NIGHTLY included) logs under subsystem com.cmuxterm.app.next, categories daemon and app.daemon. The connection's "connection lost" and "reconnect attempt N failed" lines are .info, which the system keeps only in memory, and a failed server ensure was not logged at all, so log show found nothing for this notice afterwards. Each failed start now logs a notice-level line: cmux-tui start failed (N in a row, retrying quietly|down): <detail>. Look it up with log show --predicate 'subsystem == "com.cmuxterm.app.next" AND category == "app.daemon"' --info.

This is supervisor logic in the app's Swift daemon service, where the reconnect and DaemonRecovery already live; nothing moves to Rust here.

Tests

  • DaemonRecoveryNoticePolicyTests (new): brief failures stay quiet; 3 failures over 2 min count as down; two long failures are not enough; a full disk counts at once; a working start resets the window; EnsureOutcome counts repeated identical failures.
  • DaemonRecoveryNoticeTests.aRestartThatFailsBrieflyShowsNoNoticeAndOneThatKeepsFailingDoes: the real service and notice with a stand-in daemon that dies and a stand-in cmux-tui whose ensure fails. No toast after ā‰„3 failed restarts inside the window; a toast after the window. On base the toast shows at once. The existing full-disk test still expects an immediate notice.
  • Red first: commit 5de1f1d8 (tests) and then 1a46fd4e (fix).

Observed / Not run

  • Ran the pre-push gate: no new violations.
  • Swift suites run on the fleet through CI on this PR (cmux-ci run --os macos ... CMUX_SWIFT_TEST_SUITES=DaemonRecoveryNoticePolicyTests,DaemonRecoveryNoticeTests); the result goes in a comment.
  • Not run: a live repro on the loaded Mac. Its app log for 05:15-05:22Z is requested.

Changelog

Fixed: cmux no longer shows "cmux-tui stopped and could not start again" while it is still restarting cmux-tui by itself; the notice shows only once starts keep failing.

Agent session: agent session

šŸ¤– Generated with Claude Code