cmux-tui: agents in terminals hold at most 48 of the daemon's 64 connections

AU
5d ago
+309-19

Fixes https://github.com/manaflow-ai/cmux/issues/27698

Summary

The session daemon admits at most 64 connections, and every client shares them. With about 80 agents running in terminals, terminal-bound clients filled every slot. The daemon then refused the app's own connections with server.busy: one nightly-next session refused 55,162 connections in about 2.5 hours. The app couldn't save window state or close workspaces, and the CLI failed the same way.

Now a local-socket connection from a process in one of the session's terminals (an agent, its hooks, cmux run in a pane) also takes a terminal slot. Terminals may hold at most 48 (MAX_TERMINAL_CONNECTIONS, 3/4 of 64). The next terminal-bound connection reads the same server.busy line and gives its slot back, so the other 16 stay for the app and its other local clients.

How a terminal peer is found (server/terminal_sessions.rs, TerminalSessions):

  • The accept loop, after claiming a slot, matches the peer's process session (getsid) against the session's own terminals: the process_id() of surfaces and terminal_catalog, the set the request-origin rules use.
  • A peer in the last set read answers at once.
  • A miss reads the terminals again, with Mux::try_with_state (StateMutex::try_lock, now available outside tests). While another thread holds the state lock, it answers from the last read, so the accept loop never waits on it.
  • No other process is read. It works for login(1)-started macOS shells, adopted terminals and held terminals.

A terminal refusal moves the connection from accepted to refused in server-stats. Remote-entry and WebSocket connections are unchanged.

Testing

  • terminal_connections_leave_slots_for_the_app (server/tests/outbound_and_scheduler.rs): fills the terminal share; one more terminal connection is refused and releases its slot; non-terminal connections still get the rest. It also checks the accepted and refused counts.
  • terminal_sessions::tests: malformed keys aren't terminal-bound. With the state lock held, a hit answers from the last read and a miss doesn't wait; a free lock reads again.

Hosted runs (./scripts/verify-cmux-tui-hosted.sh --filter …):

CommitFiltertest (linux)macOS
048e26e210cc (test only)terminal_connections_leave_slots_for_the_appfailure (red), run 40977996955queued
344942f566c8 (head)terminal_connections_leave_slots_for_the_appsuccess, run 40979418663 or 40979417257queued
344942f566c8 (head)terminal_sessions::testssuccess, run 40979418663 or 40979417257queued

Observed / Not run

  • Observed: the nightly-next server-stats numbers (limit 64, peak 64, accepted 15,144, refused 55,162).
  • Not run:
    • macOS hosted test jobs: queued at the time of writing.
    • No live burst against a tagged build.
    • No socket-level test drives a real terminal peer through the accept loop.
  • Not covered: WebSocket and remote-entry connections; helpers that call setsid into their own session; Windows (no session id, so nothing is capped there).

Review (claude-subagent): SHIP at 344942f566c8fbbfe4748f92fb98aa4ffb9c6dcc. The four review rounds and their resolutions are in the PR comments.

Changelog

Fixed: Agents running in terminals can no longer use up every daemon connection and lock cmux NEXT out of its own session (workspaces would not close).

🤖 Generated with Claude Code