#20376 fix(cloud): recover wedged cmux-tui links safely

Cloud machines can remain provider-running while an old cmux-tui daemon crash-loops on orphan terminal notifications. The client then reopens cached inventory into blank interactive panes.

This change:

• bounds journal startup work and avoids replaying terminal output for the agent roster, including sealed output-only segments;
• distinguishes connect deadlines from request deadlines, probes request health on the same channel, avoids mutation replay, and force-kills only a wedged child after a bounded grace period;
• surfaces stale restored terminal reservations as a recoverable failure card while preserving browser/port restoration;
• adds a state-preserving canary upgrade path that backs up and verifies the binary, stops only the daemon child, validates socket/status/health and surviving terminal hosts, and checks rollback;
• allows legitimate large-registry startup time and fixes the executable health-fence regression fixture.

The daemon journal startup work is derived from Lawrence Chen's approach in #15153; those commits are already included here and are credited in the history. This PR does not include the unrelated checkpoint work in #20259.

Changelog

Fixed Cloud workspace opening after a wedged cmux-tui daemon, with a recoverable stale-terminal state.

Validation

• python3 scripts/verify-local.py --all
• python3 scripts/verify-local.py --only swift-syntax --swift-changed
• focused Bun image supervisor tests: 2 passed
• sh -n for guest upgrade and devbox supervisor scripts
• hosted cmux-tui verification submitted for exact head 97c72dd0a85
• live claustin canary independently verified before this code hardening: published 5a4095a8... served a socket, tree freshness was current, 34/34 terminal hosts survived, and 36 terminals remained (32 running); no live VM was changed by this PR work.

Closes #20341

---

^{Need help on this PR? Tag @codesmith-bot with what you need.}

---

Summary by cubic

Fixes Cloud workspaces opening into blank panes when the remote cmux-tui daemon crash-loops or goes unresponsive while the VM stays up. Request timeouts are now reported distinctly from connect deadlines via a new requestTimedOut link error, stale terminal inventory is never materialized as interactive panes, and the guest supervisor and canary upgrade script can replace a wedged daemon without disturbing terminal hosts or the registry. Closes #20341.

Bug Fixes

• A timed-out request probes the same channel with session.ping before retiring the link; the original request is never replayed, only a failed probe drops the connection, and teardown escalates TERM to KILL after a 5-second grace period.
• Stale restored terminal reservations surface as a recoverable failure card with retry; retry restarts the reserved pane, and browser and port projections still restore independently of terminal state.
• Registry open cost is proportional to the registry, not terminal output history: quick_check runs per state table and skips journal history, the event index backfill repairs only past its high-water mark, and agent-roster replay skips terminal.output in both active and sealed segments via a kind index and a seal-time kind map; a missing journal head starts the roster empty instead of failing startup.

New Features

• The devbox supervisor probes the daemon's local lifecycle socket and restarts only the daemon after three consecutive failures once a startup grace window elapses, sized to allow legitimate large-registry startup; the verified bake with this watchdog is now the promoted default devbox image.
• web/scripts/cloud-vm/cmux-tui-upgrade.sh now handles a missing or non-serving daemon with a canary: back up and verify the old binary, stop only the daemon child, stage the target, validate serving and surviving terminal hosts, and roll back if the target does not serve.

Summary by CodeRabbit

• Bug Fixes
  • Cloud request timeouts now check connection health before disconnecting; timed-out requests are not replayed, and failures are consistently reported as timeouts.
  • Stale cloud state is reported as a recoverable network failure during terminal attachment. Retrying restarts the reserved pane.
  • Restored browser and forwarded-port panes are reprojected independently of terminal-pane state.
  • Daemon upgrades verify service health, use a canary when the daemon is not listening, and roll back unsuccessful upgrades. The devbox supervisor restarts the daemon after repeated health-check failures.
  • Workspace startup checks registry integrity and skips terminal-output records when rebuilding roster state.
  • Shutdown forcefully terminates processes that do not stop in time.