acpmux: a peer handshake survives a watch list too large to send; a peer link's watch lists only own sessions

TE
6d ago
+343-9

What broke

A daemon's peer link never finished its handshake when the peer's session list was over the 15 MiB message limit. It failed every ~2 minutes with peer disconnected: handshake timed out after 90s and showed handshaking: true, error: null in host ls the whole time.

We hit this on 2026-10-30. The peer's _acpmux/watch {"withTools": true} reply listed 5241 sessions. Only 426 were the peer's own; the other 4815 were rows it mirrors from its own peers. With tools that came to 17,181,757 bytes, over MAX_MESSAGE_BYTES (15,728,640), so the peer answered -32000 message.too_large. A read-only probe against that daemon got the error in 2.4 s. The same watch without tools is 13,635,206 bytes and still fits.

On the client, the handshake task's link_request_on(MUX_WATCH, ...).await? ended the task without a word. The read loop never looked at the task, so it waited out the 90 s timings.handshake deadline, and the peer then logged Connection reset without closing handshake.

Fix

  • Client (peer.rs)
    • When the handshake's watch is refused with message.too_large, it asks again without tools. The link connects, and its rows lack tools, where before it never connected. This works against peers on older builds, so a client update alone unblocks the link without restarting the peer.
    • The read loop now polls the handshake task. When a step fails (refused, or failed on this link), the link ends at once with handshake failed: <the peer's error>, not a silent 90 s timeout.
  • Server (server/list_reply.rs)
    • Another daemon's link (view::a_link: served as Peer, or as Web while named acpmux-peer) now gets only this daemon's own sessions in its watch reply (localOnly).
    • The link already drops every mirrored row (peer.rs watch_sessions / is_mirrored_session), so nothing it uses goes away. On the daemon above that cuts the reply from 17 MB to about 1.1 MB.
    • Other clients (CLI, app, Web) still read every row.

Tests (red, then green on cmux-ci test)

Test commit 385deed447d is red; run https://github.com/manaflow-ai/cmux/actions/runs/41020853418:

  • peer::handshake_tests::a_watch_list_too_large_with_tools_is_read_without_them failed with Elapsed (no Connected within 5 s).
  • peer::handshake_tests::a_refused_handshake_step_ends_the_link_with_its_error failed with Elapsed (the link kept waiting).
  • server::list_reply::link_tests::a_peer_link_watch_lists_only_this_daemons_own_sessions failed with ["m1", "own-chat"] != ["own-chat"].

Fix commit 565ffb95c77 is green on all three, together with watch_order_tests and the rest of peer:: / list_reply (27 passed); run https://github.com/manaflow-ai/cmux/actions/runs/41021044500. A full cargo test -p acpmux and clippy -D warnings run follows in a comment.

Risk

Low to medium.

  • The watch retry only fires on message.too_large.
  • Ending the link on a failed handshake step changes how fast a broken link redials (at once, not after 90 s), and the existing redial backoff still applies.
  • A peer link's watch list no longer carries mirrored rows. Its client discarded them already, but a third-party client that names itself acpmux-peer would also stop seeing them.

Agent session: agent session

šŸ¤– Generated with Claude Code


View with [code]smith Autofix with [code]smith
Need help on this PR? Tag @codesmith-bot with what you need.


Summary by cubic

Fixes peer handshakes that never completed when a peer's watch list exceeded the 15 MiB message limit — the client now retries the watch without tools, and a peer link's watch list now only carries the daemon's own sessions.

  • On a message.too_large refusal, the handshake asks for the watch again without tools so the link connects instead of silently timing out. This works against older peers, so a client-side update alone unblocks the link.
  • A handshake step the peer refuses now ends the link immediately with the peer's error instead of a silent 90 s timeout.
  • The server restricts a peer link's watch reply to this daemon's own sessions (localOnly), dropping mirrored rows; on the affected daemon the reply shrank from ~17 MB to ~1.1 MB.
  • All other clients keep reading every row. A third-party client naming itself acpmux-peer would also stop seeing mirrored rows, which its peer link discarded anyway.