PR #22811 - manaflow-ai/cmux

Summary

The cmux-tui daemon wrote about 21-28 MB/s to disk while agent terminals sat idle. macOS disk-write reports on 2026-10-20 name cmux NIGHTLY.app/Contents/Resources/bin/cmux: pid 53035 wrote 8.6 GB in 403 s, and pid 99026 wrote 34 GB in 1208 s. pid 99026 is cmux-tui --headless --session cmux-app (cmux-tui is a symlink to cmux). The heaviest stacks are sqlite3_step -> pwrite and sqlite3_wal_checkpoint_v2 -> pwrite.

Root cause, measured on the live daemon (read only):

sample puts all writes on the mux-session-journal-writer thread. About 47% of that thread's samples are in WAL autocheckpoint pwrite, and 20% are in F_FULLFSYNC.
server-stats journal_writer over 20 s showed 1999 batches and 2201 terminal events. That is about 100 commits/s, batch_size p50 1, and commit_us p50 5 ms.
The records are terminal.output: 43-85 byte cursor redraws (ESC[?2026h ... ESC[94;3H ESC[?25h ESC[?2026l) at about 40 Hz from each of three agent TUIs. One such record carries about 1.2 KiB of metadata, six journal_subject_index rows and one journal_event_index row.
The writer drains whatever is queued and commits at once. merge_output joined a chunk only with the entry right before it in the batch, so each PTY read became one record and one fullfsync commit.

This change adds journal_ingress/coalesce.rs:

A batch that holds only terminal output waits up to 250 ms for more output. The terminal's later output then joins its earlier record in that batch. The join can skip other terminals' output, but it stops at a barrier, a durable event, or another event of the same terminal (a resize, a gap, a new generation, or an event with a receipt).
The wait ends at once for a barrier (flush_terminal), a resize, a gap, any durable-lane event, writer shutdown, 4 MiB of input (the old limit), or 512 chunks. Each pass drains the lanes before it waits, so an event already in the queue never waits for the window.
The journal keeps every byte. Each terminal's stream order and offsets do not change. The writer now makes at most 4 output-only commits per second, and each commit holds at most one record per terminal.

Trade-off: if the daemon crashes, it can lose up to 250 ms of terminal output that no caller waited on. Before, it could lose about one commit. Output behind a barrier is still durable at once.

Second fix in this PR, found later: the reads of sealed segments were slow. While the writer committed 100 times/s, the mux-session-journal-fanout thread used about 100% CPU (sqlite3_step, mostly pread), and terminals froze after a relaunch. The tailer runs archived_records_after after each commit. That query filtered on end_sequence but ordered by start_sequence, so SQLite read every one of the 5353 segment rows. On the real 41.6 GB file (read only) it took about 10 ms per call. Ordering by end_sequence gives the same order, because segments never overlap, and lets SQLite seek the unique index: about 0.02 ms per call. EXPLAIN QUERY PLAN changes from SCAN ... USING INDEX sqlite_autoindex_journal_segments_2 to SEARCH ... (end_sequence>?). The restore cursor and the reader that skips output had the same query pattern, so the fix covers them too.

Not changed here: the 41 GB registry (auto_vacuum NONE). The writer adds about 250 rows/s and segment sealing removes about 230 rows/s, so session_journal held 15.3M unsealed rows. journal_segments, journal_event_index and journal_subject_index have no retention policy, so the file never shrinks. That needs a separate retention decision.

Testing

Behavior test at the daemon socket boundary: cmux-tui/crates/cmux-tui/tests/terminal_host_recovery/output_coalescing.rs. A terminal writes the 43-byte redraw 100 times, 10 ms apart. The test reads session.journal.subscribe until terminal.exited and asserts that all 4300 bytes are present in at most 30 terminal.output records.

Command (rbx, Linux): cargo test --locked -p cmux-tui --test terminal_host_recovery redrawing_terminal_journals

Base plus test only (a097f26dfe920): FAILED, "100 redraws took 93 terminal.output records (at most 30)". The byte check before it passed.
Head (c549adf6f4ffa): ok, 2.25 s.

Observed / Not run

Observed on head c549adf6f4ffa (rbx, Linux): the test above passed. cargo test --locked -p cmux-tui-core journal gave 174 passed and 0 failed. cargo clippy --locked -p cmux-tui-core -p cmux-tui --all-targets -- -D warnings was clean. scripts/cmux-next/prepush-gate.py was clean.

Observed on head 6349c200d697 (rbx, Linux): cargo test -p cmux-tui-core journal (174 passed), segment (15 passed), the behavior test (ok) and clippy (clean). cargo test -p cmux-tui-core restore had 65 passed and 1 failed: stash_closes_browser_tabs_into_one_group_and_restores_here_then_in_a_new_workspace. That failure is a base finding: it fails in the same way at base 6d5ca52f6f5fb (a scoped projection cross-check). Query timing: read-only sqlite3 on the real registry (shown above). No behavior test covers the query speed, because a timing test would be flaky.

Not run: macOS hosted tests (CI on this PR runs them), the full cmux-tui suite, and a tagged NIGHTLY build with a disk-write measurement after the fix. The expected drop from about 100 to at most 4 commits/s comes from the design and the Linux test. It was not measured on the Mac.

Review: a correctness review subagent returned REVISE with one must-fix (a queued barrier could wait the full window because its wake was already used). The fix is c549adf6f4ffa: the hold drains before it waits. The second pass was PASS. A separate review of the query change (6349c200d697) was PASS: only the contiguity-checked seal path writes segments, so ordering by end and ordering by start are the same.

Changelog

Fixed: Idle agent terminals no longer make the cmux background service write tens of MB per second to disk

Proof

Daemon-only change with no UI. The evidence is the measurements above and the behavior test.

🤖 Generated with Claude Code