PR #35413 - manaflow-ai/cmux

Summary

R-DAEMON-STATE stage S1 step 7 (handoff/rewrites/R-DAEMON-STATE/DESIGN.md, section 3, S1.7). S1.6b (#35082), S1.6c parts 1 and 2 (#35280, #35375) and S1.6d (#35460, the input port's writer at user-interactive QoS) are on feat.

The input lane (stage S1: the person's keys read published routes and facts, never a shared lock, and go to the terminal's own input port) becomes the daemon's default. CMUX_TUI_RDS_INPUT=0 (or the app knob CMUX_NEXT_TUI_RDS_INPUT=0) keeps the old path for one nightly; S1.8 then deletes the old path and the switch. The switch is read once, when a daemon starts: a daemon that already runs keeps its lane. To fall back, set the knob on the app's launch, then restart both daemons: "Restart cmux-tui..." restarts the app's own daemon; the Chief daemon restarts with the next app build or install-key change, or when it is stopped. Daemons outside the app get the lane by default too, each with its own switch in its own environment: Cloud session hosts (they inherit CMUX_TUI_* from the cmux-host agent), the Mac server roles (cmux host run, off by default) and SSH remotes (daemon ensure in the remote environment).

rds_switch.rs: InputLaneSwitch::from_value is on unless the value is 0, false, off or no (any case, spaces ignored; review). The first switch a process reads prints one line to stderr (cmux-tui: input lane on, CMUX_TUI_RDS_INPUT unset or ... off (the old path), CMUX_TUI_RDS_INPUT="0"). A daemon that server ensure starts has its stderr on /dev/null, so that line reaches no daemon log; the arm of a running daemon is proven by its environment (ps -wwE), as the A/B sets do.
server/tests/input_lane.rs: the kept-terminal tests of the old path pin the switch off (old_path_mux), so the fallback keeps its coverage while it ships. Every other test now runs on the lane.
DaemonDiagnosticKnobs.swift and DaemonLauncher.swift: the Chief owner (cmux-chief-*) is a cmux-tui daemon too, so it also gets the lane by default; its server ensure environment now passes the app's CMUX_NEXT_TUI_RDS_INPUT on as CMUX_TUI_RDS_INPUT, as it passes the R-LIFECYCLE keepers switch, so the fallback switch turns the lane off in both daemons (review of afadf634c63; test theInputLaneSwitchReachesTheChiefOwner, red first).
Lands after two fixes the typing A/B found (both on feat): #35765 (a terminal's output readers no longer make durable commits under the workspace registry lock; the echo stall of s17d, on both arms) and #35872 (a person's key to an idle input port is written on its own turn, with no writer-thread hand-off; the ON daemon-part rise of s17fa). This PR's own diff did not change for them: it merges their landed heads.
Prerequisites (DESIGN S1.5 bullet), in the first commit fbafa36255f: an_activity_subscriber_hears_the_persons_keys_on_both_arms and the_sizing_note_skip_sets_the_grids_the_old_path_sets (latest, smallest, largest and fixed policies; Mac and phone views; a relayed sub-view).

Why now: the S1 typing A/B (STATUS 11230)

Build 2 (S1.1-S1.6a), cmux21, OFF/ON/OFF/ON, typing into a traced terminal under the L6 load. The daemon part of each key is from the app's socket submit to the host's PTY write, in ms:

arm	p50	p95	p99	max	keys over 4 ms
OFF	0.11 / 0.10	0.56 / 0.49	1.44 / 2.19	14.14 / 11.23	5 / 3
ON	0.13 / 0.12	0.65 / 0.65	2.29 / 1.56	3.50 / 3.75	0 / 0

The OFF keys over 4 ms waited in their own send handler or in the reader (the shared-lock waits that S1 removes). The earlier "ON does not meet A4" (STATUS 10595) came from the test's echo program, which wrote a log line before each echo under the 15 GB memory load (STATUS 11019). It was not the daemon.

Caveat: the L1 load in those rounds reached 108.9-112.4 calls/s, under the 114/s rule of STATUS 7761. The kit then ran 12 L1 workers (cp/typing-under-load-rds-env@ecfb1344e4a); no worker count reaches 114/s, because the calls are lost in the load's own stalls (STATUS 11712), so each comparison stays paired (the same achieved rate in both arms).

Build 3b (S1.1-S1.6b, fleet job f1c1f3b55615af21b3259d6a, STATUS 11804), cmux21, OFF/ON/OFF/ON, L1 111.6-115.4 calls/s. Daemon part ms p50/p95/p99/max: OFF 0.10/0.57/1.37/4.59 and 0.10/0.54/1.59/3.83; ON 0.14/0.80/2.23/5.70 and 0.13/0.69/2.30/5.09. Both arms meet A4 (p99 <= 4 ms, max <= 16 ms). In this set ON's tail is about 1 ms longer than OFF's (build 2 had the opposite); the 4 ON keys over 4 ms waited after daemon.send.written (1.21-3.90 ms to the host's PTY write) or before the daemon's reader (0.33-2.80 ms), not in the dispatcher and not in their own send. The movies were lost in the fetch, so that set has no video verdict.

Set s17a (rounds STATUS 12043, 12078, 12112, 12175; round 1 video 12203; verdict 12206: ON meets A4 for the daemon key part, OFF does not), build f8685e3d (S1.7 on S1.6c, without the S1.6d writer QoS), cmux21, OFF/ON/OFF/ON, kit cp/typing-under-load-rds-env@f96caecda8d, arms proven in the daemon's ps -wwE (CMUX_TUI_RDS_INPUT=0 / =1). Daemon part ms:

round	arm	p50	p95	p99	max	keys over 4 ms (where they waited)	video key-to-photon refreshes p95 / max	L1 calls/s
s17a-1	OFF	0.10	0.66	1.81	18.7	2 (own send 18.47, 7.59 ms)	4 / 12	115.4
s17a-2	ON	0.13	0.62	1.95	9.92	1 (before the reader 5.62 ms)	4 / 40	112.6
s17a-3	OFF	0.11	0.57	2.51	23.6	3 (own send 23.45, 13.59 ms)	4 / 14	110.9
s17a-4	ON	0.14	0.72	2.45	5.17	1 (before the reader 2.27, to host 2.88 ms)	4 / 17	113.4

A4 (p99 <= 4 ms, max <= 16 ms) holds in both ON rounds and fails on max in both OFF rounds; the OFF keys over 16 ms waited in their own send (the shared-lock waits S1 removes). Keys lost 0, merged 0, order ok in all four. Video p95 is 4 refreshes in all four rounds. The video numbers come from the in-app judge with impossible frames (R-HARNESS item), so the L6 gate counts no round; the video max comes from single keys outside the daemon part (the first Ctrl-C at 12-16 refreshes in every round, one key after a Return at 40).

Set s17b (build 9d7979eb, S1.7 with S1.6d, cmux21, OFF/ON/OFF/ON; STATUS 12377): ON meets A4 p99 in both rounds (2.34, 1.45 ms) but one ON key took 27.8 ms (max A4 <= 16 ms); OFF failed max in s17b-3 (16.9 ms, in its own send). In all 4 ON rounds of s17a and s17b the echo's way back stalled (host.pty.read -> daemon.output.host gaps over 20 ms, up to 586 ms; the OFF rounds' maximum was 15.2 ms), and in 3 of them the video-max key was on that path. The video max of s17b-4 (key 248) was not; its cause is not known (STATUS 12418).

Set s17d (diagnostic hops, STATUS 12505-12559) found that stall on both arms: each terminal's output reader made durable commits (cwd, progress, program status, command facts, "last output" every 5 s) under the workspace registry lock, which waits up to 1.5 s under L6, and the next frame (the echo) waited behind it. All 16 echo gaps over 20 ms began inside one of those commits. #35765 moves the commits to a per-mux publisher thread.

Set s17fa (build 5cd1899b, with #35765; STATUS 12857, 12866, 12885): 0 echo gaps over 20 ms in all 4 rounds (echo gap max per round 2.33-3.03 ms), video not worse with ON. But ON was worse by both rules below in three places: the daemon part at p95 (+0.17 ms) and p99 (+0.53 ms), key-to-commit p99 (+1.13 ms) and echo gap max (+0.09 ms). The whole difference was in one segment, daemon.send.written -> host.pty.write: the input port's writer-thread wake (ON 0.038-0.046/0.27-0.28/0.67-0.77 ms p50/p95/p99, OFF 0.016-0.019/0.14-0.17/0.38-0.48 ms). #35872 removes that hand-off for a key to an idle port.

Set s17g at the final code (HEAD build 2ed758c77964169233240990 at b6b0309d0b7: this PR with #35765 and #35872; the PR head differs from it in feat merges, test hooks, comments, a poisoned-lock recovery on the output reader's publication check (#35765 a35e02157d2; no timing effect) and the Chief owner's input lane switch (above)), cmux21, OFF/ON/OFF/ON, arms proven in the daemon's ps -wwE:

round (cmux21, build 2ed758c7)	arm	daemon part ms p50/p95/p99/max	A4 (p99 <= 4, max <= 16)	keys over 4 ms	written -> host write p99 ms	write call p50 ms	key-to-commit ms p50/p95/p99/max	lost / merged / order
s17ga-1-off	OFF	0.110/0.582/1.490/3.123	met	0	0.413	0.005	2.81/6.70/9.72/18.17	0 / 0 / ok
s17ga-2-on	ON	0.116/0.643/1.742/2.789	met	0	0.365	0.019	2.69/6.52/9.88/14.26	0 / 0 / ok
s17ga-3-off	OFF	0.101/0.608/2.014/8.778	met	2 (8.78, 4.56; both in the old path's own send)	0.529	0.005	2.72/8.73/11.69/29.01	0 / 0 / ok
s17ga-4-on	ON	0.12/0.56/1.25/3.96	met	0	0.382	0.020	2.79/6.20/10.34/20.50	0 / 0 / ok

Pooled (STATUS 13245), ms p50/p95/p99/max:

layer	OFF	ON	ON - OFF	OFF round spread	worse by the rule below
daemon part	0.10/0.61/1.79/8.78	0.12/0.60/1.46/3.96	+0.016/-0.008/-0.331/-4.818	0.009/0.026/0.524/5.655	no
key-to-commit	2.76/8.44/10.37/29.01	2.72/6.35/9.91/20.50	-0.038/-2.087/-0.459/-8.507	0.088/2.037/1.960/10.844	no
echo gap	0.03/0.26/0.85/6.14	0.03/0.28/0.98/2.27	+<0.001/+0.011/+0.124/-3.868	0.002/0.015/0.388/3.746	no
video refreshes (all judged keys; STATUS 13303)	3/4/4/17	3/4/4/16	0/0/0/-1	1/1/1/5 (floor 1)	no

Verdict rule (coordinator decision cmux-ad m15, 2026-11-05): a layer is worse with ON only when the difference is larger than OFF's own round-to-round spread at that statistic AND larger than 0.05 ms (1 refresh for video). By that rule no layer is worse: ON meets A4 in both rounds (daemon part p99 1.74 and 1.25 ms, max 2.79 and 3.96 ms; 0 keys over 4 ms), and ON is lower than OFF at daemon part p95, p99 and max and at every key-to-commit statistic. Three statistics are higher in both ON rounds, each under the rule: daemon part p50 +0.016 ms (OFF spread 0.009 ms, under the 0.05 ms floor), echo gap p95 +0.011 ms (spread 0.015 ms) and echo gap p99 +0.124 ms (spread 0.388 ms). Its cause is known: the inline key write's own cost on the key's request thread (write call 0.019-0.020 ms at p50 against 0.005 ms for the old path's direct write; on one boundary for both arms, arrival -> socket, +0.018 ms; STATUS 13302). An earlier call at 00:42Z (STATUS 13254) kept the lane off on that p50 with a rule that had no floor; the coordinator withdrew that rule (STATUS 13355: "The rule that kept S1 OFF was mine, not A4"). The cost is addressed in its own PR, #35988 (one kernel event queue per connection for the macOS room check instead of a poll() per key, no room check on Linux, one sendmsg with no heap copy; red first), which does not gate this one.

The L6 typing pair (typing-load-l6, film and video judge) was held: the 00:42Z call had failed on these builds, and the pair would have held cmux21 for about 2 h. It runs on the next builds (set s17h, with #35988) and is reported here when it ends; it does not gate this PR.

After this change, an A/B's OFF arm must set CMUX_NEXT_TUI_RDS_INPUT=0, because unset now means ON.

Testing

Focused command: ./scripts/verify-cmux-tui-hosted.sh --filter the_input_lane_is_on_unless_the_environment_turns_it_off (Linux test (linux)).

Red at 9674a79fd8f (test/rds-s17-red-never-merge), run https://github.com/manaflow-ai/cmux/actions/runs/41795221251: rds_switch::tests::the_input_lane_is_on_unless_the_environment_turns_it_off FAILED ("the input lane is off with no CMUX_TUI_RDS_INPUT").
Green at ba4800debcc, run https://github.com/manaflow-ai/cmux/actions/runs/41795228874: passed.
Prerequisites at fbafa36255f: run https://github.com/manaflow-ai/cmux/actions/runs/41791397095 (--filter rds_input_lane), 13 passed.
The off spellings (review): red at 5fab0a55b51 (test/rds-s17-offvalues-red-never-merge), run https://github.com/manaflow-ai/cmux/actions/runs/41839881020: false_off_and_no_turn_the_input_lane_off_too FAILED ("CMUX_TUI_RDS_INPUT="false" did not turn the input lane off"); green at 6bfdf887197, run https://github.com/manaflow-ai/cmux/actions/runs/41843132208.
The old-path tests a_tab_created_or_closed_elsewhere_keeps_the_key_off_the_session_lock and a_mutation_that_routes_nothing_keeps_the_key_off_the_session_lock fail when many tests share one cargo-test process (the kept terminal's route generations are process-wide slots), on the lane-off base 01af275609c too (run https://github.com/manaflow-ai/cmux/actions/runs/41842572836); one process per test (nextest, fleet step 5f4aaea7e0ed2ff3381b4fac) passes all 25 input_lane tests at the S1.7 code. S1.8 deletes them with the kept terminal.
The full Rust suite on Linux (fleet, cargo nextest, one process per test) at the S1.7 code d62017f9c28, once with the lane on (6bfdf887197, hq run https://github.com/manaflow-ai/cmuxterm-hq/actions/runs/41846243904) and once with it off (never-merge commit b0fd51e12fe, an unset switch means off; hq run https://github.com/manaflow-ai/cmuxterm-hq/actions/runs/41846204089): 16930 tests each, 31 failed each, 29 the same (base findings: acpmux, agent-pane policy, cmux-tui cloud help and CLI docs, link_seed, a terminal_host_recovery creation test, the panic-ratchet crates). Only with the lane on: title_changed_event_includes_authoritative_surface_title (a test-only check in tree_json.rs reads the live title twice while the PTY sets it) and wait_for_off_dispatch_refuses_over_capacity_without_queueing (it also failed on the lane-off base before); each passed alone 3 of 3. Only with the lane off: 2. No test of the key path failed on either arm, so the old path keeps its coverage for the fallback night. An earlier pair of runs at d6d7b689071 against its base ca738232f32 gave the same picture (18 and 20 failed, 16 the same).
The registry latency test a_persons_requests_answer_while_busy_sessions_saturate_the_registry (it failed once in a loaded lane-on run), alone 5 times per arm on fleet Macs at the S1.7 code: lane off failed 2 of 5 (the person's new-conversation-tab p95 232 and 207 ms), lane on passed 5 of 5. The route-snapshot copy the lane adds per route change does not make the person's requests slower there.
The full Rust suite on Linux at 91c7515d459 (this PR with #35765 and #35872 at their landed heads, the lane on by default; fleet step d9e3172e4d7fb19346e8f71c): 17067 tests, 7 failed. 6 failed again alone, all base failures seen on every branch tonight: 3 acpmux tests (peer_integration, peer_request_deadlines, session_pool; acpmux does not depend on cmux-tui-core) and the 3 server::agent_transfer::tests. 1 failed in the full run and passed alone: new_terminal_first_text a_new_terminals_first_text_and_reply_do_not_wait_for_the_disk (814 ms to the first text; 770 ms of it in the new terminal host's own record write, its file data barrier before Ready in terminal_host_runtime/shared/records.rs, which no PR in this stack changes; it passed in every other full run tonight). Every key-path test passed: surface::tests::hosted 39, input_port_inline 4, output_publications 4, terminal_publisher 4, server::tests::rds_input_lane 13, server::tests::input_lane 7, rds_switch::tests 2.
The full Rust suite on macOS at 91c7515d459 (fleet step e7740de8f33233fac49a30a4): 16814 tests, 13 failed. 11 are known base failures from every full macOS run tonight (5 acpmux tests, the 3 server::agent_transfer::tests, the 2 non-UTF-8 secret-name tests, the registry latency load flake). 2 were new tonight and pass alone: pty_mouse_tracking_forwards_click_release_and_wheel_with_shift_override (cmux-tui app; 2 of 2 at afadf634c63, steps 27ece3f3483747061b4f36da and c0ee9ea9f9d2fbb456d72dec; 2 of 2 on feat cd5abc2bce7) and a_relative_diff_path_reaches_the_app_as_the_callers_folder (ENOENT when it started the binary under load; passes alone at afadf634c63, step 31cc6666d5a2f00960df2cfe). Every key-path test passed on macOS.
The Chief owner's switch (review of afadf634c63), Swift, focused command cmux-ci run --class light --script scripts/ci/package-test-lane.sh --arg=suite --arg=Packages/macOS/CmuxNext --arg=DaemonDiagnosticKnobsTests --ref SHA: red at ff229500567 (step 2a7624e2ad01d780bf7d4640: theInputLaneSwitchReachesTheChiefOwner failed, (environment → ["HOME": "/h"]) == ["HOME": "/h", "CMUX_TUI_RDS_INPUT": "0"]); green at c8900c4a230 (step c7d3b11c540a61e9eb90a08a, 6 of 6). The delta has no Rust change.
Real-app evidence: the s17ga typing journeys above (4 rounds on cmux21, the release app with this PR, all pass, arms proven in the daemon's environment). The title names no P0 item, so no separate display or smoke run applies. The What's New highlight (release-notes/next/highlights/typing-never-waits-for-busy-sessions.md) passes scripts/whats-new/validate.py in a digest of this commit.

Clippy -D warnings at afadf634c63: passed (step b8ecf9e713de6c613a7227f4). Input-lane tests at afadf634c63 on Linux: 23 of 23 pass (step c1cc883a2ce006bcf5ae3c13: server::tests::rds_input_lane 13, server::tests::input_lane 7, rds_switch::tests 2, one app projection test). Compile: fleet job 2e28e23636f6ea77a63b8b28 at the head f69479fcb8f: CmuxNext, app and cmux-tui passed (earlier: dbb0f716613fcf33dffe40ea at c8900c4a230, a5751411fd1d2f016328b734 at afadf634c63, 7c5a59d26a66d232a6888af8 at 4fe76186631, 2f7c791354ca674647a40f4e at 6bfdf887197, 3d25868d6ebd83c5ff8fa03a at 0678016998d).

Observed / Not run

Observed: the runs, fleet steps and A/B sets above. lint (linux) is red only on the base finding acpmux: clippy::indexing_slicing 687 > baseline 686 (a crate this PR does not touch).
Not run: no full Rust suite ran at the exact head. The last full suites ran at 91c7515d459 (Linux and macOS, above), whose cmux-tui-core source differs from the head only in feat changes to agent transfer and incognito history code; at the head ran clippy, the compile check, the 23 input-lane and switch tests, the two macOS reruns and the Chief switch test. The CI Rust test lanes were skipped at the head. Not run on Windows: Windows terminals get the lane by default without the inline key write (it is Unix only); test (windows) was skipped. Video: judged for s17ga (above); the L6 pair is not run yet (above).

Changelog

Changed: typing into a terminal no longer waits for the daemon's shared locks while scripts, the CLI or agents use the session.

Proof

No capture: a daemon change with no UI. The typing A/B table above is the evidence.