PR #26255 - manaflow-ai/cmux

Asked: "Fix it in a PR with before and after CPU numbers from the same measurement. Every timer or watcher needs a stop condition."

Problem

cosmic-gray's idle acpmux daemon averaged 54% of a core over 5 minutes (samples 19.5-96.4%). Profiling put 88-94% of busy samples in answering _acpmux/watch. The callers were three cmux-tui servers' agent chat notice watchers (cmux-tui-core/src/mux/agent_chat_notices.rs), each reconnecting about once a second:

follow() read the watch reply through take(MAX_LINE_BYTES = 4 MiB). The unpaged reply there is 14.7 MB (2786 sessions), so the read stopped with no newline and follow() returned Ok(()).
watch() treated that as a success, reset its backoff and reconnected 1 s later, asking for the whole list again. Each full list costs the daemon about 0.16 s of CPU.

Fix

Paged list, no cap. The watcher asks _acpmux/watch with limit: 500, then follows nextCursor with _acpmux/sessions {limit, cursor}; both are existing protocol (hub/session_pages.rs). It reads whole lines with no size cap; the daemon already bounds every message. No size cap is added anywhere.
Failure keeps the backoff. follow() returns Ok only when the list arrived whole and the connection then stayed up 60 s. A cut line, a non-JSON line, an error reply, a cursor that is not a string, or an early end is Err. watch() (unchanged) then keeps doubling from 1 s up to its 10 s ceiling instead of resetting.
Pushes vs pages. acpmux turns the watch on before it reads the page, and sends pushes separately from replies, so a push can arrive before an older page. For the rest of that connection, a chat a push reached follows only its pushes, and later page rows for it are skipped. If a push is the first thing seen about an unknown chat and the chat needs nobody, the push does the same daemon-start read of old notices that the page row would have done. A connection that ends mid-list keeps what the watch knew about chats no page reached.
Daemon: session_list_reply builds {sessions, nextCursor} by moving the list into the map instead of json!, which deep-copied it. The reply bytes are the same.
Stop conditions: the watcher thread is the existing one and still ends when the daemon shuts down or the Mux drops. This PR adds no new timer or thread.

Before / after CPU (same harness, same job)

Linux 32 vCPU cmux-ci job (run 40524468052). Release acpmux daemon with 2800 seeded idle local sessions; an unpaged list is 14,514,719 bytes, like cosmic-gray's 14.7 MB. Three copies of the real watch() thread (three cmux-tui servers) ran on its socket. Daemon CPU comes from /proc/PID/stat over 300 s, after a 30 s warm-up. Base = feat-cmux-next 577aabacc, after = this PR. The only difference between the two runs is the two changed files.

	daemon CPU, 300 s, 3 watchers	30 s samples	daemon RSS
before	63.0% of a core	56.9-79.8%	818 MB
after	0.00%	0.0 in all 10	256 MB

The same daemon with no watchers used 0.00% on both sides.

Tests (cmux-ci, Linux)

New tests in agent_chat_notices.rs, using a fake acpmux on a socket pair:

a_list_past_four_mib_is_read_whole_and_an_at_once_end_is_no_success
a_line_cut_short_is_a_failure
an_unreadable_line_or_cursor_is_a_failure
the_list_comes_a_page_at_a_time
a_connection_that_ends_mid_list_keeps_what_the_watch_knew
a_page_older_than_a_push_before_it_does_not_undo_the_push
Base with only the new tests: 6 failed, 5 passed (the existing ones).
This PR: 11 passed. acpmux hub::session_pages 4 passed. cargo clippy -p cmux-tui-core -p acpmux --all-targets -D warnings is clean.

Review

Independent Codex (gpt-6-astra, xhigh) review. The first two rounds found the push/page ordering race, malformed frames that stalled the connection, the 60 s timer being measured from connect, and the daemon-start read for chats a push reached first; all are fixed above. Verdict SHIP at a2983487688aded94f17e2bf108227e57f7e385a.

Risk

Medium. It changes how the notice watcher syncs with acpmux: paged instead of one reply, and pushes win over older pages. Against an older daemon that ignores limit, the first reply has no nextCursor, so it counts as the whole list. That works the same as before, minus the 4 MiB cut.

Overlap

Open #23190 rewrites watch() (socket-watch hints). This PR leaves watch() alone and changes only follow(), so the two compose; #23190's connect_and_follow resets on follow(...).is_ok(), which now means a whole list plus a connection that stayed up.
Open #25607 adds a Chat field and a test in the same file; there is no textual overlap with this diff.
Open #23051 also removes a per-reply copy in rpc.rs (to_line cloning the result). That second copy is left to #23051.

Agent session: agent session
🤖 Generated with Claude Code

^{Need help on this PR? Tag @codesmith-bot with what you need.}