perf(cmux-tui): the inline key write checks room with one kernel queue per connection and sends without a copy (#35413 s17ga)

Merged
LA
2d ago
+321-40

Summary

The s17ga typing A/B for #35413 kept the S1 input lane off (STATUS 13245, 13254, 13302). The daemon part was worse with the lane on at p50: +0.016 ms against an OFF round-to-round spread of 0.009 ms, with both ON rounds above both OFF rounds. On one boundary for both arms (from the key's arrival at the daemon to its bytes on the host socket), the difference stays: +0.018 ms at p50. It is all in the key's write call on its own request thread: 0.019-0.020 ms at p50 with the inline write (#35872), against 0.005 ms for the old path's direct write. Both arms write the host socket on the request thread, so this is the inline write's own cost.

The inline write did three things the old direct write did not:

  1. poll(POLLOUT, 0) before the send. On macOS it is needed: there a send with MSG_DONTWAIT on a blocking stream socket still waits for room (#35872). But XNU's poll() opens a kernel event queue, registers a knote on the socket under the socket's lock, and frees both again on every call. Freeing the queue walks every knote of the process: the fd knote lists under the process's fd lock, then every hashed knote under the hash lock. The daemon holds one hashed EVFILT_PROC knote per host child, so this cost grows with the session's terminals.
  2. Two heap copies of the key (Frame::new(payload.to_vec()), then encode_frame).
  3. One send of the frame it had just built.

Now (terminal_host_runtime/shared/attachment/room_check.rs):

  • macOS: each host connection keeps one kernel event queue, with the socket's EVFILT_WRITE registered once, at its first key. A key reads it with kevent64(KEVENT_FLAG_IMMEDIATE): one call, no allocation, no sleep (a zero-timeout poll() also slept about 10 us when there was no room). The filter's data is the socket's free room; the key goes inline only when that holds the whole frame. EV_EOF or EV_ERROR is no room, so the key goes to the port and its reconnect path. A failed kqueue() falls back to poll(). The connection's attachment owns the queue and shares it with every route clone; it closes with the attachment (a reconnect makes a new attachment and a new queue). Closing the socket removes the registration, so a stale queue never reports room.
  • Linux: no check. MSG_DONTWAIT is honoured there, and a Unix stream socket takes a frame this small whole or not at all.
  • Every Unix target: the frame header is built on the stack (terminal_host_protocol::input_frame_header, pinned byte for byte to encode_frame's output by a test). The header and the key go out in one sendmsg of two iovecs, with no heap copy of the key.

The order rules, the 1 KB size limit and every fallback to the port are unchanged. The room check still runs under the connection's writer guard, so no other daemon writer can take the room before the send.

Testing

Focused command: cmux-ci run --class isolated --script scripts/ci/cmux-tui-rust-check.sh --os macos|linux --arg test --arg input_port_inline --ref SHA.

Observed / Not run

  • Observed: the runs above, and a never-merge micro-benchmark on an idle fleet Mac (step 8ee692ca8659f49c9c816176; ns per call, p50). poll(POLLOUT, 0) 375, kevent64(IMMEDIATE) on a registered queue 167, Frame::new + encode_frame 42, a 33-byte send 1625. On an idle Mac this saves about 0.25 us per key. The same benchmark with 14 spinning threads and the measuring thread at user-interactive QoS (before the fix: step b9f9e486b09b70b51035d3b2; after: step fde1d207ce50ee8d1d22a4fc; different minis, so compare within a run): poll 416 ns, kevent64 208 ns at p50 in both runs. These benchmarks did not reproduce the A/B's 0.014 ms write-call difference: in the test process the lane-on write_bytes_replying is not slower than the lane-off one at p50 in either run. A cost that the test process does not have (never-merge macOS bench, step 9e2ae6e0d493356babb1a379): a zero-timeout poll() costs more the more kernel event registrations the process holds, because XNU's poll builds and tears down a kernel queue against the process's file table on every call. p50 375 ns with no other registrations, 1083 ns with 1024, 3584 ns with 4096. kevent64(IMMEDIATE) on a queue registered once stays at 208 ns at every count. The daemon's own count, read from the live app daemon under the L6 load (see below), is 264-266, not thousands; the benches below are at that count. In the after run, the tight 20,000-key loop sent most keys through the port (lane-on p50 83 ns: a queue push, no syscall), because the writer outran the reader and the port then stays busy; the paced test above shows that keys with room go inline.
  • Observed: the typing A/B s17ha on HEAD build e8291edc70fd5a9064248bd8 (this PR plus #35413), cmux21, OFF/ON/OFF/ON (STATUS 13405): the ON write call fell to 0.007 ms at p50 (OFF 0.005 ms; 0.019-0.020 ms before this PR), and stayed there in the 4 rounds of s17hb (STATUS 13496); daemon part and key-to-commit not worse by either rule. One flag from s17ha: echo gap max (one key per round) was higher in both ON rounds, 3.98 and 3.37 ms against 2.22 and 1.94 ms. The replication set s17hb (4 more rounds) settled it (STATUS 13496, 8 rounds): the OFF rounds reached 3.87 and 3.88 ms, pooled ON - OFF at max is +1.61 ms inside an OFF spread of 1.94 ms, and no layer is worse by the rule with the floor or the rule with no floor.
  • Observed: the daemon's real kernel event count, read without changes from the live app daemon under the L6 load (cmux21, build e8291edc, lskq and proc_pidinfo; rounds s17hb-1-off at 02:44Z and s17hp-head-1 at 03:20Z, the same in both): 264-266 knotes. 13-14 are on fds (highest fd 67-79). 250-253 are hashed, and 234-237 of them are EVFILT_PROC, one per host child, so the count grows with the session's terminals. 1052 open fds (768 sockets), highest fd 1095. The chief daemon holds 29. The daemon under that load runs 390-560 threads, 365k-388k Unix calls and 142k-149k context switches a second, at about 180% CPU (s17hp-head-2 at 03:41Z).
  • Observed: the knote bench rebuilt at that count in the test process (never-merge, step 4a530efa4a2276db2caa564e, p50 ns). poll(POLLOUT, 0): 791 warm, 6500 cold (the caches swept before each call). kevent64(IMMEDIATE): 208 warm, 1750 cold. So at the daemon's count the old check cost +0.58 us warm and +4.75 us cold per key. A knote list grown to the highest fd changes nothing (792 ns): the hashed knotes set the cost. With 1000 hashed knotes: +1.96 us warm, +23.9 us cold, so the old cost grows with the session size. The same shape with paced socket and EVFILT_USER threads at about the daemon's rates (398k Unix calls and 262k switches a second; step e7934de9e7d37a98dcf3461b): +1.58 us warm, +5.38 us cold; poll's p95 rose to 11.2 us against 1.5 us for kevent64.
  • Observed: the whole inline write before and after this PR, end to end, at that knote shape and under that paced load, with a host thread that reads the socket (never-merge, step 582752f33f1c775cf20b9cc3, 1000 keys per case, p50 / p95 ns). With the caches swept before each call (between two of the typist's keys the daemon makes about 50k calls): before 12583 / 24083, after 4584 / 9667, so +8.0 / +14.4 us per key. The room check is +5.3 / +12.5 us of that, the build and send (two copies and send against a stack header and sendmsg) +2.75 / +2.6 us. Warm (200 us apart): +2.0 / +8.2 us. Idle and cold: +5.9 / +6.3 us.
  • So, at the daemon's real knote count and at about its call rates, the bench explains about two thirds of what the A/B removed from the write call (12-13 us at p50, 20-22 us at p95), and the room check is most of it. The rest, about 4-5 us at p50, has no measured cause. The A/B's write call also holds work the bench does not have (the port lock, the route load, the flow records), and the real daemon runs 390-560 threads against the bench's 50.
  • Not run: the full cmux-tui suite at the head, and Windows (the inline write is Unix only; Windows does not change).

Performance

metric (s17 A/B, L6 load, cmux21)before (s17ga, build 2ed758c7)after (s17h, build e8291edc)harness
write call p50/p95/p99 ms, ON0.019-0.020 / 0.030-0.032 / 0.042-0.045 (OFF 0.005 / 0.008 / 0.019-0.020)0.007 / 0.009-0.011 / 0.017-0.020 (OFF 0.005 / 0.007-0.009 / 0.015-0.017), s17ha + s17hb, 4 rounds each (STATUS 13496)segments.py on the flow files
daemon part p50 ms, ON - OFF+0.016 (OFF spread 0.009)+0.006 (OFF spread 0.017), s17ha + s17hb (8 rounds)ab_summary.py, STATUS 13245, 13496
poll(POLLOUT, 0) vs kevent64(IMMEDIATE) per key, p50 ns, against the process's kernel event registrations (0 / 1024 / 4096)poll 375 / 1083 / 3584kevent64 208 / 208 / 208never-merge macOS bench, step 9e2ae6e0d493356babb1a379
the same at the live daemon's count (264 knotes, 250 hashed, 1041 fds), p50 ns, warm / coldpoll 791 / 6500kevent64 208 / 1750never-merge macOS bench, step 4a530efa4a2276db2caa564e
the same under paced load at about the daemon's call rates (398k calls/s, 262k switches/s), p50 / p95 ns, warmpoll 2000 / 11208kevent64 417 / 1458never-merge macOS bench, step e7934de9e7d37a98dcf3461b
whole inline write (room check, build, send), at the daemon's count and paced load, caches cold, p50 / p95 ns12583 / 240834584 / 9667never-merge macOS bench, step 582752f33f1c775cf20b9cc3

Counted on the live daemon: 264-266 knotes (250-253 hashed). At that count the bench gives +0.58 us warm and +4.75 us cold per key for the old check (step 4a530efa4a2276db2caa564e), about a third of the 12-13 us the write call lost. The whole old write against the new one, at that count and at about the daemon's call rates, with the caches cold: +8.0 us at p50 and +14.4 us at p95 (step 582752f33f1c775cf20b9cc3), about two thirds of what the A/B removed (12-13 / 20-22 us). The old room check alone is +5.3 / +12.5 us of that, and it grows with the hashed knotes (one per host child): +23.9 us cold at 1000. The rest has no measured cause.

Changelog

none (the input lane is on by default since #35413, which has the What's New highlight; this removes about 8-13 us per key on macOS, which no one can see)

Proof

No capture: daemon-only change with no UI. The typing A/B is reported on #35413.