fix(cmux-tui): the reap grace holds at most 16 closed terminals; past it the longest-closed ends at once (STATUS 7998)

Merged
LA
3d ago
+84-3

Refs STATUS 7998 (VIDEO EXTRA: 50 tabs closed at once kept all 208 /dev/ptmx handles 10 s later and released them by 60 s; at the L6 storm's close rate, closed-but-held PTYs reach the machine's 511 in about 48 s).

Cause

What keeps a closed terminal's host process and its PTYs alive 10-60 s after the close is the reap grace (mux/terminal_reap.rs; the app starts the daemon with --terminal-reap-grace-seconds 30, DaemonLauncher.terminalReapGraceSeconds). A closed tab's terminal keeps its shell for the grace so the close can be undone with the live shell. A shell cannot outlive its PTY, so "keep the process but release its PTY" is not possible: the PTY goes when the shell ends. Measured on the fleet (#32838's diag step b01dd8303d952fd412d5199f): each terminal holds 4 handles (3 in its host, 1 daemon custody copy), all for one PTY, and all come back when the grace ends. #32838 (merged) ends those terminals only when free PTYs fall under 64, which a storm passes faster than a reclaim pass.

Fix

The reap grace holds at most MAX_TERMINALS_IN_REAP_GRACE (16) closed terminals. On each reaper pass (a topology change wakes it), the closed terminals past the cap, longest closed first, are reaped at once through the reaper's own guarded close (screen archived as at any reap; never a kept terminal or one a client is attached to). A recent close stays undoable with its live shell. At the L6 storm's rate (about 5 closes a second) the closed-but-held PTYs stay at 16 instead of about 150.

Tests

mux::terminal_reap::tests::closed_terminals_past_the_grace_cap_end_at_once (18 workspaces, closed in a burst inside the grace, reaper after each close): the 2 longest-closed end, the 16 recent stay running, a placed terminal stays.

  • Red at ec04445def16 (test only, cap constant present but no logic): step ab09b157bbb11cd3ebfc7dd6, left: [] (nothing reaped inside the grace).
  • Green at 99a11c62b75: terminal_reap 15/15 (6e61198b30d403d5cd9eb30e), mux:: 561/561 (ec1ad3cf9f845906cccd7371; the first mux:: step got no Blacksmith run). At a1d0156da8e, pty_re 10/10 (82f02d601d92727c5028bbcf) and fmt passed (f05cb8eb6fabb67d2a95f5c9). The cap test there failed on a base defect (a create right after any reap trips the projection cross-check, #33059, repro step 0ff771ada37aaf789ff0a381), so the test now makes every workspace before it closes and reaps.
  • Clippy: red on the base in acpmux/src/hub/additional_dirs.rs:99 (step 9ecd84819970ac4e5443dc59 on feat 1114b7785f8), fixed by #33064; not this PR.

Observed / Not run

Observed: the red step and the per-process PTY counts in #32838.

Not run: the L6 storm journey again (needs VIDEO EXTRA's kit); a GUI run.

Decision

The cap value, 16: lead decision (a script burst is the case that hit the PTY limit; 16 undoable live closes is enough for a person). A close past it is reopened from its saved screen with a new shell.

Changelog

  • Fixed: cmux-next no longer runs out of terminals ("PTY capacity exhausted") when a script or agent opens and closes many tabs in a minute: at most 16 recently closed terminals keep their shell.

Autofix with [code]smith
Open in [code]smith · Need help on this PR? Tag @codesmith-bot with what you need.

Summary by cubic

Fixes PTY exhaustion when a script or agent closes many tabs in a burst: the reap grace now holds at most 16 closed terminals instead of letting every closed tab keep its shell (and so its PTY) for the full grace. Past the cap, the longest-closed terminals are reaped at once, with the screen archived as at any reap; a recent close stays undoable with its live shell.

  • At the old behavior, about 5 closes per second held ~150 extra PTYs and hit the machine's 511-terminal limit in about 48 seconds.
  • Adds a test that closes 18 terminals in a burst and verifies the 2 longest-closed end while the 16 recent stay running.