PR #34537 - manaflow-ai/cmux

Leo asked how the file/folder syncing works and why it takes so long:

how are we doing the file/folder syncage? ... that makes it take a super long time... what if it's not on GitHub?

and approved the plan: "3. Send several files at once, and send git's packs whole instead of loose files." This PR is the packs half; sending several files at once follows on top of the zstd write-stream PR.

Leo-ask: ask-6e29ca080a

What changes

cmux link sync push (the streamed copy behind Move to Cloud and a chat's copy) sent every loose object in .git/objects/xx/ as its own fs.write: a repository with a few thousand loose objects cost a few thousand round trips. Now:

The loose objects local::collect lists (the repository's own and any it borrows through alternates) are packed with git pack-objects from their names, into this Mac's state folder (<state>/link-sync/<record key>.packs/, next to the sync record this Mac already keeps). The pack and its .idx go into .git/objects/pack/ on the machine in place of the loose files. Nothing in the user's repository is written: no gc, no repack, no change to their .git.
Existing packs (and every other .git file) go as they are, whole, under the same allow-list and skip rules as before. Hooks still never go, and the secrets, config cleaning and alternates rules are untouched (they are applied before packing, and only objects already being sent get packed).
The packs made for a folder are kept and reused while any of their objects are still loose here. A re-push of an unchanged repository makes no pack and writes nothing (same pack name, same SHA-256 in the manifest). A push after a commit sends one small pack of the new objects only. After a local gc the kept packs drop out, and the manifest's delete step removes them from the machine.
If git can't pack (a broken loose file, say), those objects go loose as before.
Pull: git_back::copy_objects already skips pack files whose manifest revision matches the push, and pull.rs never treats .git as tree files, so the made pack is never brought back as a new file or a conflict. The new test checks this.

New code: cmux-tui/crates/cmux-tui/src/link/sync_cli/packs.rs (it reads the pack's v2 .idx with checked slicing and no indexing). There's one call in push_inner, gated to git copies that keep a git record (not cmux link access shares).

Tests

loose_objects_go_as_a_pack_and_the_copy_is_a_valid_repository (tests/link_sync.rs, the real cmux-tui link sync push against the real FsService): a dirty repo with one pack plus 40+ loose objects. It checks that this Mac's objects are byte-identical after the push, the machine has no loose objects, the local pack arrived whole plus exactly one made pack, git fsck --full passes, and git status, HEAD and the stash match. A re-push writes 0, and a pull right after writes 0 and leaves this Mac's objects alone. After a commit it sends one small new pack (the old one stays), then fsck and status are checked again. After a local gc the made packs go away and the copy still passes fsck.
The test helper's git now passes gc.auto=0: background auto-gc during a test was repacking objects in the middle of a push.
loose_object_push_bench (#[ignore], same file): a 3,000-object repo pushed through an in-process fs server that adds a per-request round trip (CMUX_SYNC_BENCH_RTT_MS, default 20). It counts requests, fs.writes and bytes sent.
Fleet Linux (cmux-ci test): cargo fmt --check clean. link_sync passes 19/19 (1 ignored), cloud_move_journey and link_seed pass, and the sync_cli lib tests pass. cargo clippy -p cmux-tui --all-targets -D warnings reports only two findings, both already on base (cli/automation/teleport.rs:130, cli/automation.rs:98), and none in this diff.

Performance

Same harness and env for both: loose_object_push_bench, 300 working files over 10 commits = 3,030 loose objects, fleet Linux pool. Before is feat-cmux-next at d7d34e6c86c, after is this head.

metric	before	after	delta	harness or run link
push wall time, 20 ms RTT	154.2 s	13.7 s	-91%	`loose_object_push_bench`, `CMUX_SYNC_BENCH_RTT_MS=20`
push wall time, 0 ms RTT	757 ms	254 ms	-66%	same, `CMUX_SYNC_BENCH_RTT_MS=0`
requests on the wire	7,482	655	-91%	same
`fs.write`s (files sent)	3,346	312	-91%	same
bytes sent	5.10 MB	0.87 MB	-83%	same (base64 lines as sent)
objects on the machine	3,030 loose	3,030 in 1 pack		`git count-objects -v` on both sides

Most of the time left after this is the 300 working files, one round trip each. Sending several files at once is the follow-up.

Risk

A push now runs git pack-objects on this Mac, reading the repository and writing only to the state folder. Its cost is about one compression pass over the loose objects.
The push still deletes first, then writes. On a re-push over a copy from before this change, the old loose files are deleted before the pack lands, so a push cut off in that window leaves the machine's copy short of objects until the next push resumes it. A copy is unfinished until its push ends anyway.
Two pushes of the same folder to the same machine folder at the same moment share one kept-packs folder. Neither corrupts the other's pack (each pack is made in a temp folder and renamed), but one can prune a pack the other is about to send, and that push fails with a read error and can be retried.

Agent session: agent session

🤖 Generated with Claude Code

^{Open in [code]smith · Need help on this PR? Tag @codesmith-bot with what you need.}

Summary by cubic

cmux link sync push now sends git's loose objects as one pack instead of one fs.write per object, cutting push time by ~91% on a link with round-trip latency and ~66% locally while reducing requests and bytes sent by over 80%.

What changed

Loose objects are packed with git pack-objects from the object names alone, written only to this Mac's state folder; nothing in the user's repository changes (no gc, no repack). Git runs through cmux_process so the pack obeys the same sandbox as the rest of the sync.
Existing packs and all other .git files go as before, under the same allow-list, skip, hook, secret, config, and alternates rules.
Packs are kept and reused: an unchanged repository sends nothing, a commit sends one small pack of only the new objects, and after a local gc the made packs drop out and are removed from the machine.
A made pack is always owner-only (0600): it may hold objects that were private here before they were deleted, so it stays private even when reused after its objects become public.
Objects git can't pack (broken loose files) go loose as before; pulls never bring the made pack back as a file or conflict.

Tests and performance

New tests cover a dirty repository with a pack plus 40+ loose objects, verifying the copy passes git fsck, matches status/HEAD/stash, re-push writes 0, and post-commit pushes send one small pack; a separate test checks a made pack arrives 0600 even when reused after its objects become public.
The ignored benchmark uses a 3,000-object repo through an in-process fs server with a per-request round trip (CMUX_SYNC_BENCH_RTT_MS, default 20); results below are from fleet Linux.

metric	before	after	delta
push wall time, 20 ms RTT	154.2 s	13.7 s	-91%
push wall time, 0 ms RTT	757 ms	254 ms	-66%
requests on the wire	7,482	655	-91%
`fs.write`s	3,346	312	-91%
bytes sent	5.10 MB	0.87 MB	-83%

Risk

A push now runs git pack-objects, reading the repository and writing only to the state folder; its cost is about one compression pass over the loose objects.
Two simultaneous pushes of the same folder to the same machine folder share the kept-packs folder; one can prune a pack the other is about to send, failing that push with a read error (retryable).