P0 typing under load (F4): the person's keys never wait for the daemon's session lock
Summary
P0 TYPING UNDER LOAD, finding F4. In the filmed L6 round (all loads at once), each key's delay splits into two parts. The app part (NSEvent to the socket submit to the daemon) was p50 3.8 ms, p95 9.2 ms, max 34.5 ms. The daemon part (socket submit to the byte arriving at the PTY) was p50 11.9 ms, p95 53.9 ms, p99 103.3 ms, max 475 ms. So the 250-400 ms freezes on the film are the key waiting in the daemon before its PTY write. Data: run feat-51b2e1d-probe-tul-20261102T053502Z-cmux20, probe.csv plus echo.jsonl (STATUS 5299).
Cause: a key's send looked its terminal up through Mux::surface, which takes the global session lock (Mux.state), before the PTY write. Its sizing note (note_terminal_input) looked it up again after the write, which delays the next key on the same connection. Automation holds that lock for every list, snapshot and durable mutation:
- Resource projections (
tab.list,workspace.list,session.snapshot) build the whole public snapshot under it. - Topology mutations hold it across their registry commits (
lock_state_pinned).
So the person's typing waited behind agents, the CLI and MCP.
Fix: the person's keys never take the session lock, and a kept terminal is used only while its routing is current.
- An attached client (the app's terminal view) keeps, from its first key on, the terminal its keys go to in its attach record (
AttachedSurface::input,ClientRegistry::attached_input_surface/remember_attached_input). The handle goes with the record at the last detach and at disconnect, and a dead terminal is looked up again. note_terminal_inputtakes the terminal the bytes were written to.- The typing path reads only the client registry, the terminal's own runtime lock and the sizing lock.
- Unattached senders, such as the CLI's
terminal write, keep the lookup. State::surfacesandState::terminal_catalogareRouteTables (cmux-tui-core/src/route_table.rs): reads deref to the map, and every insert or remove moves a route generation, so the compiler finds every routing write. A kept terminal carries the generation read before its lookup; a respawn, close, move, projection or rollback makes the next key look it up again.
This follows Chrome's model: the person's input goes before automation, on its own path.
Testing
Regression tests (cmux-tui-core server::tests::input_lane), command scripts/ci/cmux-tui-rust-check.sh test input_lane (macOS, fleet):
a_key_reaches_its_terminal_while_automation_holds_the_session_lock: another thread holds Mux.state for 300 ms; a key from an attached client must take under 150 ms. RED on the base (55564d1e71b1, step 4531ad570dba6876c215b0bd): the key waited 310 ms.after_a_respawn_the_next_key_reaches_the_new_terminal(lead order: the kept terminal must never outlive its routing): a new runtime takes the placement id while the old one still lives; the next key must reach the new shell. RED on the first kept-terminal cache (branch fix/p0-typing-f4-respawn-red b8891a169635, step 81b3ea940914c7d231014290): "the key after the respawn did not reach the new shell".a_mutation_that_routes_nothing_keeps_the_key_off_the_session_lock(review): a pane rename and a dropped state clone must not send the next key back to the lock. RED at e306e3578ed5 (a Drop that moved the generation on every clone, step 6c9c6954e23ef09e4c6c3752).
GREEN at the head: input_lane 4/4 (step 16e1b7bc0a472cf31844ed80); macOS server:: 1454 tests at 8bd3d0a (step 3ed23f3eb02487c33c689a4e): all pass except connection_handler_owns_every_router_connection_operation, the base red fixed by #31939. clippy -D warnings (macOS): passes at 84470f980daf for this PR's code; at the head it fails only in cmux-tui/src/cli/app/tests.rs and cmux-tui/src/link/mod.rs, files from the feat merge (#28950), not this PR.
Full macOS test run at 8bd3d0a (step 902245a674203ade9f94de40) against the same step on the merge base 219d9bda0fca (step 63a5b91299b92b6ab12d9bf9): 30 and 26 failures. 20 test names failed only in the head run. Each one passes when run alone at the head 15c133d (step, filter, result):
| Failed only in the head full run | Alone at 15c133d |
|---|---|
acpmux chats_activity a_chat_carries_its_live_sessions_attention_and_latest_reply | c8d8478ab48ed320113ae69f: 1/1 pass |
acpmux chats_open a_chat_in_a_store_no_profile_uses_adopts_with_that_store, an_opencode_chat_opens_by_adopting_it_over_acp, chat_open_then_session_new_adopts_from_the_profile_store | a5313cd6a60d8a79a89f09b3 (filter adopt): 171/171 pass |
acpmux chats_settings with_discovery_off_only_the_listed_roots_are_read | cb2decc170e8ef455f0d3f5b: 1/1 pass |
acpmux durable_agents stalled_turn::an_adopted_turn_whose_model_went_quiet_ends_as_stalled_and_frees_its_agent | 5a61c7afbc3b8aa4e9e9ce3f (filter stalled_turn): 9/9 pass |
acpmux hub_integration session_git_state::git_state_of_a_held_chat_follows_a_checkout_mid_session | 0a3c489cc626834f901b858b: 11/11 pass |
acpmux router_lifetime router_exits_when_its_daemon_is_killed | f949724544341bdbbebc2330: 1/1 pass |
cmux-link caller::supervised::tests::supervised_link_accepts_only_the_live_child_and_enables_its_stamp_policy, ..._rejects_a_matching_pid_with_a_different_start_instance | 0202439eff3d685dcb76ea5d (filter supervised): 7/7 pass |
cmux-tui app::tests::pty_mouse::pty_mouse_uses_the_canonical_rendered_grid_during_resize_margins | 80e8e0ff5c36f035325684aa: 1/1 pass |
cmux-tui link::e2e_tests:: (4 tests) | 881b1e137d9989803df2781b: 6/6 pass |
cmux-tui link::host_inbound::native_token_tests::native_token_receiver_daemon_stamps_backend_identity_and_signed_time | 3f86ead88f07c049f3c5e213: 18/18 pass |
cmux-tui-core shell_integration::incognito_history_tests:: (2 tests) | 98731f5b5d9cb7d8300663ee: 17/17 pass |
cmux-tui-core surface::tests::child_process::default_shell_prompt_survives_rapid_resizes_after_a_partial_line | 8aa14526280ca05b3b3c5723: 1/1 pass |
cmux-tui-core surface::tests::lifecycle::macos_surface_spawn_returns_within_deadline | 8a826f9a8edfaaeac4fec7a3: 1/1 pass |
None of them is in the changed code. The head full run took 3791 s and the base run 1708 s on a loaded fleet, and that run reported 13 leaked processes; these are timing and load failures of the full run. An order dependence cannot be fully ruled out by single runs; no failing test touches the input lane.
Observed / Not run
- Observed: the field split above (my analysis of the measurement owner's round), and the fleet steps above.
- Not run yet: the measurement owner's L1 and L6 rounds on a build with this change (socket submit to PTY arrival, in-app NSEvent to echo frame, keys merged). The PR is a P0 fix, so those numbers go here when they arrive.
- Read only: CPU starvation of default-QoS daemon threads under L6 may be a second part of the daemon wait. If the after-numbers still show waits, a follow-up raises the attachment threads to
QOS_CLASS_USER_INTERACTIVE.
Changelog
Fixed: Typing in a terminal no longer freezes while agents, the CLI or MCP list, read or change the session
Proof
No capture: daemon-only change, no visual change.
Follow-ups (review, non-blocking): the generation is global, so a tab create or close anywhere sends every client's next key through the lock once (a per-id check can follow); refresh_host_policy_locked -> workspace_default_of still takes Mux.state per key when a workspace size default is set; send_key still looks up through Mux.state. F7 (daemon thread QoS) is a separate finding.