fix(cmux-tui): a kill of everything restores each terminal's exact text, then the session line, in the same directory string (REOPEN-F3)

Merged
LA
4d ago
+2172-103

Summary

REOPEN-F3 (P0, data loss; STATUS 4421, film reopen/dogfood-f350103/kill-daemons-light/a1-011556/reopen-1.mp4): after a kill -9 of the app, the daemon, the holder and every terminal host, each reopened terminal showed only — session restored (previous process ended) — and a new prompt. The last screen (400 numbered lines) and the 363-row scrollback were gone, and the directory read /private/tmp instead of /tmp.

Lawrence's rule (cmux-ad 01:30Z): when the terminal hosts die, the stored output restores the text exactly, then a clear "session ended" line, and the new shell starts in the same path string.

Cause 1: the text was lost

The output store's default mode (snapshot, from #22937) wrote a terminal's screen at most every 30 s after its first new output. A kill of the daemon and its hosts in that window found no stored screen, so the respawn seeded only the marker line.

Fix: a tail journal with the snapshot as its compaction (cmux-tui-core/src/output_store/tail.rs):

  • The output after the last snapshot is appended to <terminal>.tail by the store's background writer every 100 ms (write(2) only; the PTY path stays memory-only). Each tail is fsynced at most once per second.

  • The snapshot records the stream offset its screen ends at. The offset is read under the parser lock while no parsed chunk is in flight (the journal capture epoch is even), so the tail continues the snapshot byte for byte. When the snapshot is in place, the tail file goes.

  • A snapshot is written for a terminal that this daemon has no snapshot of (a new generation), after a resize, at most every 30 s while the content changes, when the tail passes 1 MiB, when 1 MiB of unchanged redraws waits in memory, and at checkpoint, clear-history and quit.

  • Restore: the snapshot replay plus the tail bytes from exactly its offset, rebuilt in a scratch terminal of the snapshot's size. A gap or another generation means the snapshot alone.

  • Cursor redraws (same content digest) still write nothing: the idle tests pass in every mode.

  • The memory budget pass of a write pass runs at most once per snapshot interval, as before (the growth trigger covers the time between).

  • Worst-case loss: the output of the last 100 ms plus one writer pass when every process dies (kill -9 of all, a crash). Bytes that change nothing the content digest sees (text, size, active screen, history, cursor position; for example a cursor redrawn in place or a color-only change) wait in memory up to 1 MiB, so an idle TUI writes nothing; a kill can lose those too. An OS crash or a power loss can add up to 1 s (the fsync interval). The loss does not depend on the output stopping. The snapshot still holds the screen plus 2,000 lines (unchanged limit); full mode is unchanged.

  • A disk that refuses writes: each terminal backs off its snapshot tries (1 s, doubling to 30 s; quit and journal checkpoints still try). Bytes that cannot be stored are dropped; the snapshot at the end of the backoff covers them. See the failure-mode table.

  • Disk cost: about 16-25 KB per second per pane while a full-screen TUI repaints its whole screen at 50-60 Hz (table below); 0 bytes for an idle terminal or a cursor-only redraw.

Cause 2: /tmp became /private/tmp

The PTY child got no PWD for its cwd. A shell keeps an inherited PWD only when it names its working directory, else it shows getcwd(3), the real path. cmux-pty now sets PWD to the cwd path string (absolute paths only). A caller's own PWD, set or removed, wins.

Failure-mode table (snapshot mode, per terminal)

One rule bounds every row: when unwritten bytes cannot continue the tail, they are dropped (also right in append, past 8 MiB), and while the log still needs a snapshot, every pass asks for the next one at the earliest time one may be written (100 ms; after the backoff when the disk fails; after the size has held 1 s after a resize), whether or not the terminal prints again. That snapshot is taken from the live terminal, so it covers the dropped bytes' effect on the screen and on its recent history. Nothing waits in memory for it. A snapshot holds the screen plus about 2,000 lines of scrollback (the unchanged snapshot limit: a VT replay of up to (2,000 + rows) x cols x 3 bytes, 256 KiB to 4 MiB). A restore is that snapshot plus the tail after it. Full mode keeps every byte in segment files; it is unchanged, apart from row 1.

inputmemorydiskworst-case loss on a kill of all processestest (red on the base of the fix)
normal output, a kill at any timeunwritten bytes of one pass (100 ms)tail at most 1 MiB raw plus one append (compacted into a snapshot at 1 MiB or 30 s); snapshot up to 4 MiB gzipthe last 100 ms plus one writer passkill_all_restores_the_exact_screen_and_scrollback_then_the_session_line, kill_all_while_output_streams_restores_every_line_but_the_last_moment
flood (the writer falls behind)at most 8 MiB unwritten, checked on every PTY read (append; the writer is woken at 4 MiB), then dropped, plus the writer's in-flight copy (for output that does not compress, up to about 3 x 8 MiB for a moment, one terminal at a time); a drop while a snapshot is written still asks for the next oneeach append at most 8 MiB rawthe last 100 ms plus one pass; past 8 MiB unwritten, the snapshot at the next pass covers it (screen plus about 2,000 lines)a_flood_past_the_cap_falls_back_to_the_snapshot, a_flood_during_a_backoff_keeps_at_most_the_cap_in_memory, a_drop_during_a_snapshot_write_still_asks_for_a_snapshot; full mode: a_flood_flush_writes_bounded_frames_and_asks_for_the_next_pass_at_once
idle TUI (cursor-only redraws)held redraws at most 1 MiB, then dropped and a snapshot at the next pass0 bytesnothing (the content did not change)idle_redrawing_terminals_write_nothing_in_snapshot_mode, a_redraw_of_exactly_the_stored_bytes_waits_in_memory, held_redraws_past_the_idle_cap_fall_back_to_the_snapshot
a chunk in flight at the captureas normalas normalas normal (never held as a redraw)a_chunk_in_flight_at_the_capture_still_reaches_the_tail, bytes_after_the_capture_still_reach_the_tail
full disk / writes refusedat most 8 MiB unwritten (checked on every PTY read) plus the writer's one in-flight copy, then droppedno growtheverything since the disk refused writes, as with any store (nothing can be written); a snapshot is tried after a backoff of 1 s doubling to 30 s, and at quit and checkpointsa_failing_disk_does_not_retry_the_snapshot_on_the_next_pass, a_flood_during_a_write_backoff_does_not_wake_the_writer_per_chunk, a_flood_during_a_backoff_keeps_at_most_the_cap_in_memory, a_tail_that_does_not_continue_keeps_no_bytes
slow diskat most 8 MiB unwritten plus the writer's one in-flight copy (one terminal at a time), then droppedas normalone pass plus the slow writeas for the flood row
huge framethe 8 MiB capan append is at most 8 MiB raw; a tail stops at 32 MiB raw plus one append while snapshots fail (below the 64 MiB read limit)as for the flood rowa_flood_past_the_cap_falls_back_to_the_snapshot
unreplayable screen (a Kitty upload, never ended, larger than the replay budget)the 8 MiB captail up to 32 MiB rawthe restore keeps the text; the half-sent image is aborted in the scratch terminal. While the upload stays unfinished, once the tail cannot continue (past 32 MiB of tail, after a resize, or in a new generation), no snapshot and no tail can take more output: the loss has no bound until the program ends the upload (an edge case)an_unreplayable_screen_does_not_stop_the_tail, a_tail_that_ends_inside_an_oversized_upload_still_restores_its_text; coverage: a_screen_that_ends_inside_a_control_string_keeps_the_marker_line
resize storm (a divider drag)about 0 (dropped until the size holds)one snapshot when the size has held 1 s, not one per pass; it comes even if the terminal stays idleoutput in the 1 s after the last resize plus one passa_resize_storm_writes_one_snapshot_after_the_size_settles (at most 2 snapshots during the storm, then the snapshot at the settled size)
crash during a writeas normala cut tail frame ends the read; a snapshot is temp file, fsync, rename, directory fsync before its tail is removedthe last passtail_bytes_continue_the_snapshot_offset_and_stop_at_a_gap_or_cut_frame
corrupt or truncated tail file, another generation, a gapnonenonethe restore uses the snapshot plus the longest valid tail prefix that continues it exactly, or the snapshot alonetail_bytes_continue_the_snapshot_offset_and_stop_at_a_gap_or_cut_frame
many terminals at onceper terminal as above (normally one pass of output; at most 8 MiB while flooding)per terminal as above; idle terminals write nothingper terminal as aboveno extra test: the per-terminal tests hold per log; kill_all_heavy_load_disk_writes_per_pane is a measurement (12 panes), not a bound

Known limits, not changed here: under a constant flood every capture try can find a parsed chunk in flight; the tail then grows to its 32 MiB cap and the covering snapshot comes when the flood eases (memory and disk bounds hold). The 4 MiB writer wake has no test of its own (an improvement, not a table bound).
Other known limits: tail files are not counted in terminal.persistence.maxTotalBytes (normally at most about 1 MiB per terminal; up to 32 MiB only while snapshots fail); history beyond about 2,000 lines comes back only in full mode.

Disk cost and the alternatives (fleet, macOS 26)

12 panes, each a TUI that repaints all 40 rows with new content at about 50-60 Hz; daemon logical write bytes per pane per second (proc_pid_rusage ri_logical_writes), 8 s windows, one test at a time (step ec1a2ebc0c8fe45c01955399, NEXTEST_TEST_THREADS=1, scratch branch cp/reopen-f3-measure-matrix), two runs each:

policy (same 100 ms loss bound)scroll (1 row into history per frame)inline repaint (cursor-addressed, no scroll)alternate-screen repaint
this PR: snapshot at 30 s or at 1 MiB of raw tail22.9-24.8 KB19.5-19.9 KB15.8-19.0 KB
compact at 64 KiB of raw tail97.7-114 KB91-98 KB83-94 KB
compact when the tail is 1x the snapshot file131-166 KB211-221 KB205-209 KB
compact when the tail is 4x the snapshot file72-75 KB96 KB83-104 KB
a snapshot instead of the tail when it is cheaper24.4-26.0 KB17.0-17.8 KB19.6-22.7 KB
memory mode (nothing on disk)0-42 B0-85 B0-170 B

A snapshot of these screens is about 1.8-3 KB, the same as 100 ms of tail. So a reset at a full repaint or an earlier compaction does not remove bytes: each compaction adds a snapshot write. The current rule is the lowest or equal at the same loss bound. For 12 such panes that is about 190-300 KB/s (0.7-1.1 GB per hour) while they all repaint; #22937's problem was 13-25 MB/s while idle. An earlier 20 s run with parallel tests gave 36.9 KB (base 85 B; steps 7998d492ce9f2d53cbaea7a0, 471c39c64daa23b523b70b4c).

Tests (red on the base, green on the head)

  • kill_all_restores_the_exact_screen_and_scrollback_then_the_session_line: 400 lines in a 40-row shell, a kill of the daemon, the shell and its host 2 s later, with the production 30 s interval. The restored transcript must equal the transcript before the kill line by line, and the next line must be the session line.
  • kill_all_while_output_streams_restores_every_line_but_the_last_moment: a loop prints until the kill. Every line from the first must come back in order with no gap. At most 0.5 s of lines may be lost.
  • kill_all_respawned_shell_keeps_its_directory_path_string: a shell started in <dir>/link (a symlink) prints $PWD = <dir>/link, before and after the kill (each probe has its own label).
  • output_store::snapshot::tests: a chunk in flight at the capture, or bytes after it, still reach the tail; a redraw of exactly the stored bytes waits in memory; a failing disk does not retry the snapshot on the next pass, stops the tail and keeps the bytes; held redraw bytes stay bounded; a snapshot in any mode removes a tail file; a flood during a write backoff does not wake the writer per chunk.
  • cmux-pty spawn_sets_pwd_to_the_cwd_path_string_not_its_real_path: the child's PWD is the path string, and a caller's own PWD wins.
  • output_store::tail::tests: tail bytes continue the snapshot offset exactly and stop at a gap or a cut frame. A restore writes the tail over the snapshot screen.

Commands: cmux-ci run --class isolated --script scripts/ci/cmux-tui-rust-check.sh --ref SHA --os macos --arg test --arg <filter>.

Observed / Not run

  • Observed: the red and green runs above on fleet Macs (macOS 26), and the disk measurement.
  • Respawn time (daemon alone, step 795917bfad594f610cd2554f, scratch cp/reopen-f3-measure-latency): from the new daemon's spawn to the respawned runtime in its tab with the restored text and the session line: 1 terminal 144 ms, 4 terminals 152-153 ms, 12 terminals 180-244 ms.
  • Not run: the full cmux-tui suite, clippy and fmt (the PR's CI runs them). The real app: a recording build of this head is job 86af1b06f57a87114961ecb0 (cp/reopen-f3-recording-build, this fix plus the REOPEN VIDEO bench mapping). The REOPEN VIDEO owner runs the journey reopen-kill-all-scrollback and films the kill-all case on it. Results go in a comment.
  • Not covered here: STATUS 4480 (on a feat app build, the killed terminal is in no tab after the reopen). The daemon-only kill_all tests on feat keep the terminal in its tab under the same id, so that defect is outside this daemon path. It is under investigation and gets its own fix.

Risk

risk:high: the change adds a product-code file delete (fs::remove_file of a tail file that a newer snapshot covers). It changes no setting. It fixes a data-loss defect, so it does not ship dark. terminal.persistence.mode = memory still turns all output files off.

Changelog

Fixed: after a crash that kills cmux and all its background processes, each terminal comes back with its last screen and scrollback (up to the last 0.1 s of output), and its shell stays in the same directory path (/tmp no longer becomes /private/tmp).