PR #34538 - manaflow-ai/cmux

Leo asked:

"how are we doing the file/folder syncage? ... that makes it take a super long time... what if it's not on GitHub?"

and approved: "2. Compress what's sent and stop encoding it as text."

Leo-ask: ask-6e29ca080a

What changes

Daemon (cmux-tui-core). A new capability, fs-write-deflate-v1, is advertised only where fs-v1 is (fs_ops::advertised). It is host-dependent, so it is kept out of build_capabilities like fs-v1. Under it, fs.write takes "encoding":"deflate" with decoded_size on both the line op and the write stream. The wire bytes (the line's bytes_base64, or the stream's size raw bytes) are one raw deflate stream (RFC 1951):

The free-space check and the 12 MiB single-write limit apply to decoded_size.
Decoding is streamed into the same PendingWrite and stops at decoded_size. Decoding to more or fewer bytes, or bytes that are not deflate, is params.invalid and leaves nothing at the path, not even the temporary file. A small payload can't fill the disk.
The write stream also takes permissions, as the line op already did.
Atomicity is unchanged (temp file, fsync, rename). spec/commands.md (fs-v1) documents it. The contract fixture is unchanged: it records no stream and no new field.

Client (link sync push, single-file copy, seed bundle upload). The client asks identify once per connection. When the machine lists fs-write-deflate-v1, it deflates each part at level 3, but only if the file's first 64 KiB shrink by more than a tenth. Packs, images and archives go as they are. It also checks that each compressed part is actually smaller. Without the capability, every request is byte-for-byte the old base64 request, so old machine images keep working. Unchanged:

progress callbacks
IO_DEADLINE and the answer budget
8 MiB parts with append and rename
the resume manifest (SHA-256, permissions, revision)
the Mac-side skip rules

Round trips. The benchmark showed the push is bound by latency, not bandwidth: 490 MB at 8 MB/s needs about 61 s, but the push took 495 s. Answering each file's "what is in the way" check from the cached folder listing cut that to 308 s. Review found that this lets a symlink the machine swaps in mid-push be written through (#34362's protection), so that change was taken back out of this PR. It returns with a no-follow write in #34999.

Also fixes the stale seed.rs header: "Not wired into start-chat yet." has been false since start-chat's copy started calling seed_cli::push.

What this does not do yet

"Stop encoding it as text" over the raw write stream is not reachable on a Cloud copy today. The stream is routed only on the link's remote entry (route_first_line). The remote connect --local-socket bridge carries bounded JSON lines to the daemon's local socket, whose line path refuses "stream":true. Carrying the stream needs changes in cmux-remote: #34482. The bigger remaining win is sending files over several connections: #34483. Both are measured with the harness below.

Backward compatibility

New client to an old machine: no capability, so plain base64 exactly as before (a_machine_without_the_capability_gets_plain_base64).
Old client to a new machine: no encoding field, so the daemon writes the bytes as before.
The new fields are accepted only together. decoded_size without encoding, or any encoding other than deflate, is refused.

Tests

fs_ops::encoding unit tests: round trip, a decode past decoded_size refused (64 MiB of zeros declared as 1 MiB), short or foreign bytes refused, the line limit applied to decoded_size.
fs_wire_tests, end to end through route_first_line: a 40 MiB file over one deflate stream with 0755 kept; a decompressed-size overrun leaves no file or temporary; a line write decodes, keeps 0600, and refuses an overrun.
remote.rs: the old-machine fallback sends plain base64; with the capability, a file larger than one part decodes from its parts and random bytes go uncompressed.
tests/link_push_bench.rs: a_push_to_a_cloud_daemon_crosses_compressed_and_lands_exact, against a real Cloud-shaped daemon (exact bytes, 0755, wire bytes below the tree size).
Unchanged suites pass: link_sync (18), link_access (5), cloud_move_journey (3), sync_cli lib (21), fs_ lib (32).
Clippy -D warnings on cmux-tui-core and cmux-tui is clean in the changed code. Two base reds in cli/automation* (manual_is_multiple_of, type_complexity) are untouched by this PR.

Performance

The same harness on both sides: link_push_bench, a real headless daemon with the Cloud identity, pushed through a unix-socket proxy at 8 MB/s each way with 30 ms one-way latency. The fixture is 3,190 files (365 MB, a 172 MB .git: 3,000 source-like text files packed plus 160 MiB of random blobs left loose). "Before" ran on the commit with only the harness, 3294d76eb0c.

metric	before	after	delta	harness or run link
push wall time	494.8 s	489.4 s	-5.4 s (-1.1%)	`link_push_bench`, fleet Linux job b081eb17 (before) / run 41786488153 (after, at e9478e9)
bytes up on the wire	490.7 MB	461.0 MB	-29.7 MB (-6.1%)	same
bytes down on the wire	0.81 MB	0.82 MB	+0.01 MB	same

Measured but not shipped here: with the per-file fs.stat answered from the cached listing, the same harness gave 308.1 s / 460.6 MB (-37.7% wall). That change is #34999. The fixture is mostly incompressible random blobs, so compression's byte gain is small here; source-heavy trees gain more (the 200-file text test crosses in well under the tree size). The packs change (#34537, merged into this branch by update-branch) is not part of either number.

Reuse:

cargo test -p cmux-tui --test link_push_bench -- --ignored --nocapture link_push_bench
# env: CMUX_BENCH_FILES=3000 CMUX_BENCH_BLOB_MB=160 CMUX_BENCH_RATE_MBPS=8 CMUX_BENCH_LATENCY_MS=30

The diff is about 950 lines with tests and the benchmark harness, which another PR shares (the shipped code is about 330 lines). It is kept as one PR because the capability, its decoder and its client are one protocol change, and the benchmark's before number needed the harness on its own commit.

Agent session: agent session

🤖 Generated with Claude Code

^{Open in [code]smith · Need help on this PR? Tag @codesmith-bot with what you need.}

Summary by cubic

Speeds up Cloud copies by compressing fs.write bytes through a new fs-write-deflate-v1 capability, while machines or clients without it get plain base64 exactly as before.

The fs.write line op and write streams accept "encoding":"deflate" with decoded_size; wire bytes are one raw deflate stream, decoded as they arrive. Decoding to more or fewer bytes, or to non-deflate bytes, is params.invalid and leaves nothing at the path; the free-space check and write limit apply to decoded_size. Write streams also take permissions.
The client (link sync push, single-file copy, seed bundle upload) deflates parts at level 3 only when the machine advertises the capability and the file's first 64 KiB shrinks by more than a tenth. Without the capability or the fields, every request is byte-for-byte the old base64 one.
Reverted the cached-listing change: the "is a folder in the way?" check is again a fresh fs.stat right before each write, so base's symlink protection stays current; reusing the listing needs a no-follow write first.
Raw write streams still travel as base64 lines on Cloud copies; stream routing and multi-connection copies are follow-ups.
Fixes the stale seed.rs header, which claimed the copy path was not wired into start-chat.