PR #28685 - manaflow-ai/cmux

Summary

Leo, 2026-10-31: "why is there a 15mb cap and why are we not following protocol/transport best practices with compression/better formats".

An acpmux peer link sent its whole session list as plain JSON text: about 16.6 MB without tool inventories for the ~3,700 rows two fleet daemons list, in pages under the 15 MiB per-message cap. That cap is ours (rpc::MAX_MESSAGE_BYTES). It stays under the 16 MiB WebSocket frame that older acpmux builds read with, and it stays as the safety rail. This PR makes the bytes on the wire about 6x smaller, so a list no longer has to sit near the cap to cross a link.

What changes. The vendored tungstenite (0.29) has no permessage-deflate, so acpmux negotiates its own compression on each WebSocket (frame_compress.rs):

A peer link lists zlib in initialize _meta.acpmux.compression.
From then on, the server sends each message of 16 KiB or more as one Binary frame holding the zlib stream of its JSON text, at level 1. Smaller messages stay Text. The initialize reply says _meta.acpmux.compression: "zlib".
The peer's reader inflates Binary frames. It refuses one that inflates past the 64 MiB WebSocket limit, or that isn't zlib-wrapped UTF-8, and starts the link over.
A client that never asks gets Text exactly as before: every older build, the Mac app's pane, webviews, the CLI. A server that never heard of the field ignores it, so old and new peers work together in both directions.
Only the server compresses. A client's Binary frames are upload data (upload_frames.rs), and what a client sends in bulk (images) is compressed already.
The per-message cap still bounds the JSON text, so paging (#28182, #28542, #28588) is unchanged.
A message of 1 MiB or more is compressed on a blocking thread, which keeps the runtime free. The link's connected log line now gives frame_bytes and text_bytes, so the saving shows in the journal.

Binary encodings (CBOR or MessagePack): not worth building. On the same 3,700 rows, MessagePack is 85% of the JSON's size before compression, but only 95% after zlib (1.61 MB against 1.69 MB). zlib level 6 alone does better (1.35 MB). On top of compression, a second wire format would save under 10% and would need a codec in every client (Swift, TypeScript). Compression gets almost all of the gain.

Performance

The sample is real rows (an acpmux ls --json --all of 443 sessions, 2.7 KB a row, 4.0 KB with tools), repeated to fleet size. Copies sit 1.2 MB apart, past zlib's 32 KB window, so repeating the rows doesn't inflate the ratio: 443, 3,700 and 20,000 rows all give the same 6.0x. Times are CPython zlib on this lane VM (the same zlib/deflate level 1 that flate2 runs), single core.

metric	before	after	delta	harness or run link
handshake wire bytes, 3,700 rows, no tools	10.11 MB	1.69 MB	-83% (6.0x)	real rows x8.4, zlib level 1
handshake wire bytes, 3,700 rows, with tools	15.18 MB	2.14 MB	-86% (7.1x)	same, `--with-tools`
handshake wire bytes, 20,000 rows, no tools	54.40 MB	9.09 MB	-83%	same
handshake wire bytes, 20,000 rows, with tools	81.10 MB	11.46 MB	-86%	same
server CPU to compress, per MB of text	0	5.3 ms (181-225 MB/s)	+5.3 ms/MB	same
peer CPU to inflate, 3,700-row list	0	34-39 ms	+39 ms	same
wire time for the 16.6 MB fleet list at 50 Mbit/s	2.7 s	0.45 s	-2.2 s	bytes / link rate (computed)
20,000 synthetic rows (67.3 MB) over a loopback link	67.3 MB, 544 ms	1.24 MB, 600 ms	-98% bytes, +56 ms	`compressed_link_tests` (repetitive test text compresses far better than real rows; loopback has no wire cost)

Testing

New tests:

frame_compress::tests: initialize negotiation; a large line goes Binary only once asked, a small one stays Text; the blocking-thread path; and a frame that inflates past the limit, or isn't zlib text, is refused.
server::compressed_link_tests::a_20k_row_list_reaches_a_peer_compressed_every_row_once: a real listener mirrors 20 peers x 1,000 rows (over 50 MB of JSON). A Peer link connects, every row arrives exactly once, and the frame bytes are under a quarter of the text bytes.
server::compressed_link_tests::a_client_that_does_not_ask_gets_text_frames: a client that doesn't ask gets only Text frames and no compression in the initialize reply. One that asks gets Binary frames.
The existing peer_handshake_tests fake peers answer in Text and ignore _meta.acpmux.compression, as an old server does. They still pass, so a new peer still links to an old server.

Ran on this lane VM: cargo test -p acpmux --lib -- compressed_link peer::handshake_tests frame_compress gives 16 passed. That covers the 9 existing peer::handshake_tests (they include #28588's) and the ws_shutdown writer tests, which ran in an earlier pass at the first head. The bad-frame regression test fails without its fix: the link waited out the test's 10 s deadline (a_compressed_frame_that_is_not_json_rpc_ends_the_link ... FAILED). It passes with the fix.

Added after the first review, which flagged it: a Binary frame that inflates but isn't a JSON-RPC message now ends the link. Before, the reply in it was dropped silently and its caller waited (server::compressed_link_tests::a_compressed_frame_that_is_not_json_rpc_ends_the_link).

Observed / Not run

Observed: the tests above, on this lane VM. The size and CPU numbers come from a real 443-row list on this VM, measured with CPython zlib, which is the same level-1 deflate flate2 uses.

Not run: the full acpmux suite and clippy (CI runs them on this PR), and a live link between two fleet daemons. After this lands, the connected to ... journal line of a fleet daemon, with its frame_bytes and text_bytes, gives the live numbers.

Changelog

Changed: Peer daemons send large messages, such as the session list, compressed, about 6x smaller on the wire.

Proof

No capture: daemon transport only, nothing on screen changes.

Agent session: agent session

🤖 Generated with Claude Code

^{Need help on this PR? Tag @codesmith-bot with what you need.}

Summary by cubic

Compresses large acpmux peer messages on the wire when the peer asks for it, cutting the fleet session list (3,700 rows, ~16 MB of JSON) to about 1.7 MB.

A peer lists zlib in initialize _meta.acpmux.compression; the server then sends each message of 16 KiB or more as one Binary zlib frame at level 1 and confirms it in its initialize reply. Smaller messages stay Text.
A client that never asks gets Text exactly as before, and a server that never heard of the field ignores it, so old and new builds interoperate in both directions.
Only the server compresses; client Binary frames are upload data. The 15 MiB per-message cap still bounds the JSON text, so paging is unchanged.
Messages past 1 MiB are compressed on a blocking thread; the link's connected log now reports frame_bytes and text_bytes.
A compressed frame that inflates past the WebSocket limit or isn't JSON-RPC ends the link, instead of dropping a reply its caller waits on.