#28654 fix(cmux-next): no client clock closes a live daemon's connection (P0 v2.2.10 no new workspaces)

Summary

The app's heartbeat closed its own control connection to a live cmux-tui after two unanswered identify asks. Every request on that connection then failed with "cmux-tui connection closed: closed by client". On v2.2.10-opt under swap (19.9 of 21.5 GB used, load 13), this failed 9 workspace creates, 36 window saves and the history trail for 47 minutes. The daemon was healthy the whole time: registry wait p99 0 ms, 0 connections refused. Lawrence's report: "I can't even make new workspaces".

Now no client clock ends a live daemon's connection:

• After its misses, the heartbeat marks the daemon not answering (DaemonConnection.daemonAnswering, debug.daemon answering) and offers Restart cmux-tui (DaemonRecovery.offersRestart).
• The connection stays open. Its pending requests get their replies when the daemon answers again.
• Only a real signal ends the connection: EOF, a socket error, or the daemon process's exit. LineTransport.peerProcessIsAlive reads the kernel's peer pid (LOCAL_PEERPID) and runs kill(pid, 0).
• A bridged connection still closes after its misses, because this Mac cannot see its daemon.

This is row 1 of the timer sweep (cmux-ad, STOP THE LINE). The perf branch also needs two more fixes, which follow as their own PRs: close() must end the connection itself, and a shared terminal stream must never park the control reader.

Behavior change

Before, a SIGSTOP'd or wedged owner turned the store disconnected (cx-jniyz.4, #20955). Now the store stays connected, answering is false, and Restart is offered. The restore journey's daemon-stop variant keys on offers_restart and then Restart, so it still holds. A visible "cmux-tui is not answering, still working..." banner is a follow-up.

Testing

Focused command (fleet, CmuxNext package test lane), using python3 /work/wt/chrome-parity-design/proof-step.py:

• Red on the base: red --branch cp/fix-p0-app-connection-close-feat --ref 01bd9b05d8ff8d3021fbc5d5c7b6dd1ef9f4c9be --suite HeartbeatLiveDaemonTests -- --red. Job 72c251afcf58d71651ff017c: failed as required (cmux19-perf log: 4 issues at HeartbeatLiveDaemonTests.swift:37, 38, 44 and 46: outcome → .closed, the transport closed, "the create failed while the daemon was alive: cmux-tui connection closed: closed by client", and closes == [.closedByClient]).
• Green on the head: green --branch cp/fix-p0-app-connection-close-feat --ref 4e5ff11834ed9ff7bde506ed787b1d0de22d859b --suite HeartbeatLiveDaemonTests,StoppedOwnerTests. Job 010bb0309a5ba38c11e2944f: passed (cmux19-perf: HeartbeatLiveDaemonTests passed in 0.6 s, StoppedOwnerTests passed in 7.2 s against a live daemon; "All 2 suites passed", no FLAKY notice).

HeartbeatLiveDaemonTests uses a fake daemon thread that reads every request and answers none until it is released. StoppedOwnerTests uses a live daemon. It now expects the daemon to show "not answering" and stay connected, and a request sent while the daemon is stopped to get its answer after SIGCONT.

Observed / Not run

• Observed:
  • prepush-gate.py passed (no new violations) at every head.
  • Compile-check 27f4ecd9a09775fef3f977de of 657012a311b passed. Compile-check b2c85d0a2dc5448dc423236f of 4e5ff11834e passed (CmuxNext, app and cmux-tui receipts all ok:true).
  • The independent review of 657012a311b returned FIX (the stuck-owner auto-restart was lost). 4e5ff11834e restores it: each unanswered ask past the misses runs one plain server ensure.
  • Red 72c251af failed and green 010bb030 passed, as listed above.
  • Independent review: SHIP at 4e5ff11834ed (https://github.com/manaflow-ai/cmux/pull/28654#issuecomment-6537288745).
• Not run: the restore journey's daemon-stop variant (I only read it: it waits for offers_restart, which this PR keeps). There is no test of the full path from a connected owner that does not answer, through the ensure probe, to the automatic replace (review note 3).

Changelog

Fixed: cmux no longer drops its connection to a busy cmux-tui, so new workspaces and tabs no longer fail with "connection closed" on a loaded Mac.

No capture: no native UI change (a debug.daemon field and the existing Restart offer).