PR #35765 - manaflow-ai/cmux

Summary

Under load, a terminal's output could pause for tens to hundreds of milliseconds right after the person typed, on both input lanes. Each terminal's output reader made the durable commits its own output asked for: the shell's directory (OSC 7), progress and program status (OSC 9;4, OSC 7501), the command and prompt facts, and "last output", which it pushes every 5 s while the terminal prints. These are resource commits under the workspace registry lock. Under the L6 typing load that lock waits up to 1.5 s, and the next output frame (the echo of the key) waited in the reader's queue behind the commit. In the s17d diagnostic rounds for #35413, all 16 echo gaps over 20 ms (to 501 ms) began inside one of those commits, with the S1 input lane on and off (26 to 27 slow commits per round, about every 5.1 s). The reader also posted OSC 9/777/99 notifications and bell attention marks, which are durable commits of the same kind.

Now the reader only checks, in memory under the terminal's own locks, whether a publication is due, and queues it on a per-mux terminal publisher (mux/terminal_publisher.rs): one worker thread that runs the queue in order. A terminal has at most one publication queued, so changes that arrive while it waits are published with it as the latest state, which is what every publication commits. The reader still takes and rate-limits the notifications at the output that asked for them (OSC 9/777/99, OSC 7501 alerts, bell marks); they queue one by one in output order, each after the publication of the records it names. The new peeks (ShellActivity::has_change, TerminalMetadata::has_publication_change, ProgramStatusRecords::has_publication_change) mirror the takes they precede; take_change now uses has_change, so the two cannot drift.

All four reader sites move: hosted output, hosted resize, hosted reconnect and the local PTY reader. Adoption, the first view of an app terminal, a terminal read, the topology commits and a hosted terminal's exit still publish directly on their own threads. The exit publishes the shell's last directory before it commits, as the reader did before it handled the host's Exit frame (a publication that runs after the exit commits nothing). At shutdown, after the readers stop, the publisher finishes its queue within the same reader grace, before the journal stops.

Known and unchanged: a tab closed while its directory publication waits for the registry keeps the previous directory for reopen. The inline reader waited for the same lock as the close and lost the same race.

Testing

Red first, each focused command cmux-ci run --class isolated --script scripts/ci/cmux-tui-rust-check.sh --os linux|macos --arg test --arg FILTER --ref SHA:

output_publications (echo stall), red at 2c75845dec6 (tests only, on feat 3e464b08ff9): Linux step 10a6bce6aee2e1e9ed938504 and macOS step 3725e378b9d9359a71a439e1 fail both tests (hosted: the next frame did not reach the terminal in 2 s while another thread held the registry; local: no output in 2 s). Green at the fix a91fb3a343e: Linux step f2906bd89e6d3bf8f6e093dc, both pass in 0.09 s.
Review fixes, red at 19bcc9ec556 (tests and test-only seams on 454b2c14e7e): a_terminal_exit_commits_the_directory_its_shell_reported_last fails on Linux (step 03727fdd31cd26e015a5395d) and macOS (step b226157a6db7aeb9c55f953d): the exited terminal shows no directory instead of the shell's last report. a_programs_alert_posts_after_the_notifications_of_earlier_output fails on Linux (step 32e1980912768e1c29f2a46b): the notifications post as ["make asks a question", "first-note"]. Green at 6a64e181a1b on Linux: output_publications 3 of 3 pass (step c61bfb7cc07337f6ed0f41b6), terminal_publisher 3 of 3 pass (step 64882ed8fe0fb3b7ea72a2bb).
Host loss (second review), red on never-merge branch test/rds-pub-hostloss-red-never-merge at 22b5bca002f (the test without the fix): a_host_loss_commits_the_directory_the_shell_reported_last fails at its assertion on Linux (step f8ce430a9f57519b158bf625) and macOS (step 2195563525df91ba71cc48ed): the terminal shows no directory after the loss instead of the shell's last report. Green at e1e72888af2: output_publications 4 of 4 pass on Linux (step 795541cc23895f5912c156d2) and macOS (step 9e042aca3b9f8cbcd875c351); terminal_publisher 3 of 3 pass on Linux (step 91369408c0287b71ff2a9d9e). (The first version of this test, 3108cf0bb75, failed at its setup on both sides and proved nothing; e1e72888af2 corrects the setup.)
Full cmux-tui suite at a91fb3a343e: Linux 17037 tests, 10 failed (step 25a44d054061eb5f335fefe3); macOS 16776 tests, 12 failed (step b657d58397b3dacace07df63). None is in the changed code: every new and changed test passed on both. The 3 server::agent_transfer::tests failures ("the daemon must not reach acpmux") and interrupted_public_creation_publishes_once_and_replays_stable_ids_after_two_restarts also fail alone on the base 2c75845dec6 (steps af2dce05e377d2964d76aa3a and f8636ad308baf77f023c98f9). The acpmux failures cannot come from this change: acpmux has no dependency on cmux-tui-core or cmux-tui (Cargo.lock closure), and its tests do not start the cmux-tui binary. The macOS-only git_seed / link::seed non-UTF-8 secret name tests are base failures. The rest failed in the full run and passed alone. At 6a64e181a1b on Linux (step 2ca29db079690609b2dfd6f5): 17039 tests, 11 failed, the same set as at a91fb3a343e except mux::layout_invariants::tests::layout_ops_conserve_tabs (a known load flake of the scoped projection check on the base, 2 of 15 full runs in STATUS 12503) and server::agent_session_attach::proxy::shares::tests::an_expiry_any_path_notices_ends_the_guests_connection_and_stream (agent session shares, no terminal output path); both failed in the full run and passed alone. Every new and changed test passed. macOS at 6a64e181a1b: running.
Test-only entity check (found by the full suite at b2ebe6809df: a publisher-built TabChanged raced the reader's title in the cfg(test) check of tree_entity_json and poisoned the session): red at 2f52b7eb82e, the_entity_check_holds_when_a_reader_changes_the_title_between_its_builds fails on Linux (step 6e11c5df7e6c9b1f859c1053); green at 429c696d427 (step 9052ae5a16cedb378d1a5a1b).
At 429c696d427: clippy passed (step 1f57059c7e37d03a445db2d9); compile-check passed for CmuxNext, app and cmux-tui (job 0f3f7c166a5ee91bbf0b9bdd); full cmux-tui suite on Linux 17041 tests, 8 failed (step de82b046f3f3dca36d972379), all of them in the base set above; on macOS 16780 tests, 24 failed and 1 timed out (step 767dc2cec79c4a406e422fd0): acpmux, the base agent_transfer and interrupted_public_creation failures, the macOS non-UTF-8 seed tests, and 8 cli agent_session_start tests that timed out waiting for acpmux in the full run; all 41 agent_session_start tests pass alone at the head (step e969f1d77ffdfcfe6830c411) and on the base (step b958211321ff67af637d3a5a), so they are full-run load flakes, not this change. Every new and changed test passed on both.
Exit notifications (third review), red on never-merge branch test/rds-pub-exitnote-red-never-merge at f696021d69a (the test without the fix eff16dc8ea2/e24fcfe29fd): a_notification_queued_before_an_exit_posts_before_the_exit fails at its assertion on Linux (step bbc01c14dac96cdbbf24d18c) and macOS (step 065b1afdc8c850e8d2dc5a34): 0 notifications posted by the end of the exit, 1 expected. Green at 019ad0b88c4 on Linux (step 713381686df8f5da1c2940c2) and macOS (step 94d76158bc35a012cc092e23). (The first version of this test, 37fd891b8a3, passed on the old code and proved nothing; 019ad0b88c4 corrects it.) At e24fcfe29fd: terminal_publisher and output_publications 4 of 4 each on Linux, terminal_publisher 4 of 4 on macOS, compile-check passed (job 25c2473f79fa9e9a8ba86055); 3eede4880f2 changes only the test file after it.
Review receipts: first review SHIP at a91fb3a343e, second SHIP at 6a64e181a1b, third SHIP at 3eede4880f2 (each with findings, fixed above, red first); fourth review NO-SHIP at bf46130aab3 (two lock().unwrap() calls the crash and panic ratchets count, fixed at the head a35e02157d2, below); fifth review SHIP at the head a35e02157d2.
At 3eede4880f2: clippy -D warnings passed (step 30dee12015e5673fc813d376); terminal_publisher 4 of 4 on Linux (step dc024ef31446cc0b61271a83); compile-check at 3eede4880f2 passed for CmuxNext, app and cmux-tui (job cddfd293cdc2431ab455b5fa). Full cmux-tui suite on Linux at e24fcfe29fd (step 0fad36df6fe33b21ba61f347): 17042 tests, 11 failed; 8 failed again in the rerun alone, all in the base set above: 4 acpmux tests (dispatch_durable, peer_integration, peer_request_deadlines, session_pool; acpmux does not depend on cmux-tui-core), the 3 server::agent_transfer::tests and interrupted_public_creation_publishes_once_and_replays_stable_ids_after_two_restarts; 3 passed alone, none of them in this change's path: acpmux::web_token_rotation rotate_token_replaces_the_live_and_saved_token_and_closes_web_connections (acpmux does not depend on cmux-tui-core; it failed the same way in the full run of #35872 at ef166da6f4d, which has no publisher, and passed alone); cmux-tui::link_ensure link_stop_waits_for_the_process_and_fails_for_a_link_that_does_not_end ("stop answered before the process was gone": link stop returns after the link's exit event, and the test's kill(pid, 0) still found the pid, a process not yet reaped under load; the test writes no terminal output, and it passed in the full runs at 429c696d427 and ef166da6f4d and alone here); state::group_update_latency_tests::a_persons_requests_answer_while_busy_sessions_saturate_the_registry (the known load flake: run alone 5 times on fleet Macs with the lane off, the old path, it failed 2 of 5 in #35413; it also failed in the full run of #35872 at ef166da6f4d).
At bf46130aab3: the CI crash ratchet was red at 3eede4880f2 ("rust cmux-tui-core: expect: head 191, base ok"; fast gates runs the same ratchet): the test-only Surface::set_test_title hook used .expect( and .unwrap() outside a test module. The hook now sets the title only on a terminal surface and recovers a poisoned lock; its one caller asserts the title it reads back. crash_ratchet_compare.py is ok against the merge base 3e464b08ff9 and feat 8f4dff69776. At bf46130aab3: compile-check passed for CmuxNext, app and cmux-tui (job 95b49f55acf622362e1b83a0); clippy -D warnings passed (step 8148ed9ec54557ba5f7bd3f8); the_entity_check_holds_when_a_reader_changes_the_title_between_its_builds passed on Linux (step fcc40ef0ff0d69517520288c).
At the head a35e02157d2 (fourth review): Surface::publication_due, on the output reader's path, took the terminal's metadata with lock().unwrap(), and the test hook Mux::install_live_terminal_runtime_for_test (in mod test_hooks, which the ratchet counts) took the state the same way. Both now recover a poisoned lock (unwrap_or_else(PoisonError::into_inner)), as the program-status lock in publication_due already did; no behavior change unless a lock is poisoned. The PR now adds no .unwrap(), .expect(, panic macro or exit outside test modules (diff against the merge base). At the head: compile-check passed for CmuxNext, app and cmux-tui (job cd1f95b5ed66958f0fd31b68); clippy -D warnings passed (step b2ba344daf7c25d30d2fe040); the Rust panic ratchet (cmux-tui.yml lint step, cmux-tui-rust-check.sh panic) passed, 92 crates checked (step bc6e6e3661e2cdecbee7a9d6); output_publications 4 of 4 on Linux (step e787b11c1b2753c9136f5954); terminal_publisher 4 of 4 on Linux (step 1d084c5dd38e7cb3fabfa710).

Observed / Not run

Observed: the s17d diagnostic rounds for #35413 (STATUS 12505, 12523, 12540, 12547, 12556, 12559) that located the stall; the red and green runs above.
Observed: the s17fa A/B for #35413 on build 5cd1899bfae1ee19147b3fb5 (this PR at 454b2c14e7e), cmux21, rounds OFF/ON/OFF/ON under L6: 0 echo gaps over 20 ms in every round, echo gap max 2.86 to 3.03 ms; pooled echo gap p50/p95/p99/max OFF 0.04/0.29/0.91/2.94 ms, ON 0.04/0.28/0.88/3.03 ms (STATUS 12857, 12885).
Not run yet: the A/B on the head. Set s17g on #35413 runs on build 2ed758c77964169233240990 (b6b0309d0b7, with this PR at 3eede4880f2); the fixes after 3eede4880f2 change a test hook and two poisoned-lock paths, no key or echo path.
Not run at the head: the full cmux-tui suite (last full Linux run at e24fcfe29fd, above) and the macOS tests (last at 019ad0b88c4).

Performance

metric (L6 typing load, S1 lane off and on)	before (s17d diagnostic rounds)	after (s17fa, build 5cd1899b)	harness
echo gaps over 20 ms	16 in 4 rounds, up to 501 ms, each inside a reader's commit	0 in 4 rounds	typing-under-load kit, STATUS 12559, 12857
echo gap max per round, ms	185.7 to 501.4	2.86 to 3.03	same

Changelog

Fixed: Terminal output no longer pauses for up to half a second under heavy load while the daemon saves the terminal's directory, progress or activity.

Proof

No capture: daemon-only change with no UI. The typing A/B with movies on the fixed build is reported on #35413.