acpmux peer snapshots (5): the hub keeps a copy of what it sees of each peer session

Merged
TE
9d ago
+636-52

Lawrence's list item, word for word: "start new VMs from acp chat - decouple persistence of acp trajectories from the VM itself, in case the VM is resuming".

Part of the client-snapshot piece of manaflow-ai/cmuxterm-hq#4381 ("Persistence, pause and resume, and history hydration"): the Mac's acpmux keeps a durable, bounded copy of each peer session's events, so the app paints a paused or offline peer chat at once, marked possibly stale, and catches up from the host with afterSeq. The host's journal stays the authority; the copy is not a write lease, an execution checkpoint or a backup. The shared fenced-writer database from #4381 is not part of this work.

This replaces #25323 (store) and #25378 (hook and pane), split so each piece lands on its own under about 800 changed lines, tests included. The series:

  1. the on-disk form of a copy (#25448, landed)
  2. the store's thread: bounds, writes, removals (#25445)
    2t. the store's tests (#25505)
  3. what starts or grows a copy: generations, joins, links (#25446)
  4. purges, removed peers, the queue's bound (#25447)
  5. the hub's taps (wiring) (#25449)
  6. the read hook _acpmux/peer_snapshot, which the VM start/resume work calls too (#25473)
  7. the pane's offline banner and the reply's reading (#25474)
  8. the pane paints the copy, then catches up (#25475)
    8t. the pane's tests (#25506)
    8t2. tests of 8's attach fixes (#25620)

This PR (5): the hub keeps a copy of what it sees of each peer session

The wiring. This PR wires the store into the hub and removes the module's dead_code allow.

  • Listings and purges (peers.rs, peer_cache.rs): a peer's session list, a session gone, and a peer removed are sent to the store under the remote_sessions lock. A page asked for on a row read after them is therefore queued after them.
  • Pages (peer_forward.rs, capture_peer_page): the answers to forwarded session/attach and events requests carry the peer instance, the generation and the link epoch, all read before the request is sent.
    • An events page with no afterSeq is treated as after = 0.
    • The local attach handler reads lastSeq before the page (requests.rs).
  • Live records (peer.rs): each record a peer streams is passed along with its link. A peer's _acpmux/lagged notice names its instance and link too, so a late notice of an older link never stops a copy a newer link joined (W6).
  • pageThrough (protocol/events.rs, requests.rs, requests/events.rs): the local attach and _acpmux/events replies say how far the page holds: the log's last seq, read before the page, for the newest page or a forward page that reads to the end; for a beforeSeq page, or a forward page with more to read, the page's own last record (W22). A peer's copy covers past a page's last record only that far (3). It is a new required field in the spec, and the generated web types are regenerated.
  • Removed peers (peer_cache.rs): their snapshots go under the rows lock once the peer has left the registry. A listing that found the peer current is queued before the removal, and none comes after it.
  • Kinds. A page asked with kinds (a name or a list) that leaves out transcript is not kept.
  • Forwarded purges: they drop only the copy of that peer and of the generation the row the request was resolved by names (forget_remote(peer, mux, generation)); a row that names none removes only such a row (W21).
  • Shutdown (shutdown.rs): flush_now writes what is not yet written and waits at most 5 s for it.
  • Behaviour. The store starts filling and keeping copies under peer-snapshots/ in the daemon home, beside peers/ (never in it). Nothing reads them until the hook in 6.

Findings from the earlier reviews this piece answers

These are W-side items of S3, S14, S15, G1, G4 and G8 in the ledger:

  • blocking work on async threads;
  • a page with no cursor;
  • the attach lastSeq;
  • listings and purges ordered under the lock;
  • links sampled before the request;
  • forwarded purges limited to their peer and generation;
  • this PR's first two reviews: a removal racing a listing (W1); a page that claimed a record it did not hold, when its peer read lastSeq after the page, and the record came on a link that ended or an instance replaced since (W2, W3), answered by pageThrough;
  • this PR's fourth review: the lag notice carries its link (W6);
  • this PR's fifth review: an older watch list published after newer notices (W7, answered in 3: generations only grow) and a purge told again (G20, answered in 4). The reader-side order of the hub's own row map is fixed on feat-cmux-next by #25512 (the watch list is published before any notice read after its reply); the server-side window (a purge sent before the watch reply that carries the older list) gets its own follow-up in hub/peers.rs, a purge seen before a link's first list leaving a tombstone that list cannot revive. Neither is in this stack.
  • this PR's sixth review: an events reply now carries the journal's createdAt, and capture checks it (W8); a purge from a removed peer instance cannot remove its replacement's copy (forget_remote checks the instance under the rows lock, W9).
  • this PR's seventh review: the shutdown's peer snapshot flush is the last step of Hub::flush, after the teardown (W10); flush_now's cost and its losses are answered in 2 and 4 (W11, W12).
  • this PR's eighth review: a page goes on the link it is labelled with (Peer::request_on, W13); a page naming no generation only joins a copy, by design (W14).
  • this PR's ninth review: spawn_peer registers a replacing instance under the rows lock, so a purge or notice of the instance it replaces, checked current under that lock, lands before it (W15).
  • this PR's tenth review: a removed peer's cache, snapshots and rows go under the rows lock it leaves the registry under (W16); captures check their instance and send under that lock (W17); a page that lost records to strip_asks is not kept (W18).
  • this PR's eleventh review: a down peer's copy head not stale (W19, answered: every head is written stale by snapshot_row, and the test asserts it).
  • the lease-test failure was this PR's: the snapshot store lived in peers/.snapshots, which the test's listing of the peer caches saw under load; the store now lives in peer-snapshots/ beside peers/ (W20), with a test.
  • this PR's thirteenth review: a purge takes its generation from the row the request was resolved by, and one that names none removes no row or copy of a generation (W21); pageThrough of a beforeSeq page or a forward page with more to read is the page's own last record (W22). Tests for both; each fails without its fix.

Tests

On the fleet, the 7 new tests pass: the hub keeps its snapshots beside the peer caches, never among them; a capture of an instance replaced while it was sent is not kept; a listing under way when a peer is removed never lets its copies back; a page covers past its records only to the pageThrough its peer sends; a down peer's records and pages are kept, while purges drop them; a purge through this daemon drops only its peer's generation; and an events page with no cursor covers only what it holds. A purge that names no generation leaving a session of one, and the pages saying how far they hold (a_page_says_it_holds_to_the_log_end_only_when_it_reads_there, over _acpmux/events and _acpmux/attach), are tested too; each test fails with its fix reverted. The full cargo test -p acpmux suite passes except the base reds listed below. cargo fmt --check and cargo clippy -p acpmux --all-targets -D warnings are clean.

Release compile (cmux-next, Xcode 26.6) on the fleet through the controller at head ac100f92d2977881a1ca4be68adfb62fd7f28c43: green, step 7e2997a4966e766400ddce69.

No capture: this PR changes no UI. Its only webview file is the generated protocol types (types.ts), which gain the pageThrough field and its doc comment, and nothing in the pane reads that field yet.

Risk: medium. Every peer session's events are now copied to disk, within the bounds above. A failed write retries and never blocks the hub.

Timers, threads and loops

  • The store's thread (acpmux-peer-snapshots, Store::run), added in 2. It blocks on its channel until an op comes or the next due write or retry is reached (recv_timeout), and it ends when every sender has dropped, that is, when the hub drops. It holds no timer of its own beyond that one wait.
  • Writes go out FLUSH_AFTER (2 s) after a copy changes.
  • Retries (a write that failed, a refused removal, a root that did not read) start at 1 s and double to 60 s. Each retry stops once it succeeds, or once its copy or path is gone. A flush never cuts a retry's backoff short; only the shutdown flush tries once more.
  • flush_now (shutdown, 5) waits for its answer at most ANSWER_WITHIN (5 s), and does so through block_in_place on the multi-thread runtime.
  • No other timers, observers or polling.

Base reds, not from this PR (they fail the same way on feat-cmux-next): agent_tools::tests::own_servers_run_under_the_daemon_prefix, and the flaky session_pool::pooled_sessions_end_with_the_daemon_and_after_a_crash.

Agent session: agent session

šŸ¤– Generated with Claude Code