acpmux peer snapshots (4): purges, removed peers and the queue's bound

TE
9d ago
+622-58

Lawrence's list item, word for word: "start new VMs from acp chat - decouple persistence of acp trajectories from the VM itself, in case the VM is resuming".

Part of the client-snapshot piece of manaflow-ai/cmuxterm-hq#4381 ("Persistence, pause and resume, and history hydration"): the Mac's acpmux keeps a durable, bounded copy of each peer session's events, so the app paints a paused or offline peer chat at once, marked possibly stale, and catches up from the host with afterSeq. The host's journal stays the authority; the copy is not a write lease, an execution checkpoint or a backup. The shared fenced-writer database from #4381 is not part of this work.

This replaces #25323 (store) and #25378 (hook and pane), split so each piece lands on its own under about 800 changed lines, tests included. The series:

  1. the on-disk form of a copy (#25448, landed)
  2. the store's thread: bounds, writes, removals (#25445)
    2t. the store's tests (#25505)
  3. what starts or grows a copy: generations, joins, links (#25446, landed)
  4. purges, removed peers, the queue's bound (#25447)
  5. the hub's taps (wiring) (#25449)
  6. the read hook _acpmux/peer_snapshot, which the VM start/resume work calls too (#25473)
  7. the pane's offline banner and the reply's reading (#25474)
  8. the pane paints the copy, then catches up (#25475)
    8t. the pane's tests (#25506)
    8t2. tests of 8's attach fixes (#25620)

This PR (4): purges, removed peers and the queue's bound

Purges and removed peers:

  • A purge (Gone{generation}) bars that generation. A listing of the same or an older generation stays barred. A newer generation, or a full listing without the session, clears the bar.
  • A late purge of an older generation leaves a newer copy alone, on disk before the scan too (the bar alone decides what an earlier run kept), and a late listing of a barred generation takes nothing and leaves the newer one listed and kept.
  • A removed peer (ForgetPeer) loses its files one by one, and its directory goes only when empty. A peer that is added again keeps its new copies.
  • Before the disk is read. Purges and removals that come before the scan are recorded (unread, forgotten) and applied to what the scan finds. Each holds at most UNREAD_KEYS (4096); past it, one flag (unread_all) makes the scan drop every copy an earlier run kept (G19).

The queue's bound:

  • Every op costs at least OP_BYTES and is reserved by CAS against QUEUED_BYTES (16 MiB).
  • A capture that does not fit is a loss of its whole peer (LostPeer). The peer's copies are dropped and none is left with a hole.
  • While a peer's loss waits, nothing more of that peer is queued, and the lock is held across the check, the reservation and the send. Loss markers are therefore bounded by the number of peers, and flushes by one.
  • A removal of the peer lost into a loss is kept with it (a flag per lost peer): the loss then removes the peer whole, bars and links too (G18).
  • A capture the thread did not take (no thread then) is a loss of its peer too, and a loss not sent yet is sent again at the peer's next op, so no peer stays latched (G21).
  • A purge told again without its generation (the peer's notice after this daemon's own purge, once the row is gone) keeps the generation bar it has, so the session recreated since is listed again (G20).

Two fixes to 3's code (#25446), from the repo's own Codex review of #25446, which came after it landed (G31, G32):

  • An empty catch-up joins its link. A page after afterSeq that holds all there is up to afterSeq itself (pageThrough = afterSeq: a reconnect's catch-up that found nothing new) covers the empty run after + 1..=after and joins the copy on its link. Before, it covered nothing and returned first, so every live record after a reconnect or lag was refused until a page with records came.
  • A lag no longer turns a late page into a rollback. The copy keeps the link a page last joined it on (joined, which behind() reads) apart from whether that link's records still join (epoch, which a lag clears). Before, a lag cleared both, so a late page of the same link that read lastSeq below a record taken before the lag removed the whole copy.

Findings from the earlier reviews this piece answers

These are G6 to G12, S9 and S18 of the ledger (Codex gpt-6-astra xhigh, rounds 1 to 11 of #25323):

  • tombstones that expire, block reused ids, or are cleared by an equal listing;
  • generation-blind purges;
  • losses or purges before the scan that come back;
  • a coalesced loss that swallowed a later purge;
  • a peer directory removal that took a re-added peer's files;
  • loss markers that bypassed the reservation, one per session (round 11);
  • this PR's first review: a purge of an older generation before the scan, a late listing of a barred generation (G13, G14); flush and loss markers are bounded per peer (S30);
  • this PR's third review: a removal lost into a loss (G18), and pre-scan sets with no bound (G19);
  • this PR's fourth review and 5's fifth: a capture not sent (G21), a purge told again (G20);
  • this PR's fifth review: every op drains all unsent loss markers, not only its own peer's (G22); purge bars of unlisted keys are capped at BARRED_KEYS (G23); record seqs are answered by 3's rejection (G24);
  • this PR's sixth review: every op and flush sends every loss waiting, whichever peer's (G25, W12); a purge charges its key and bars nothing for an id no file can have (G26); an op the thread never took (it ended) is a loss of its peer (G27);
  • this PR's seventh review: a bar evicted before the disk read takes everything an earlier run kept (G28); a flush waits for every loss to go (G29); the next thread takes an ended one's untaken ops as losses before anything it is sent (S62).
  • this PR's eighth review: the bar check was read as reversed; it skips a generation at or below the bar and takes a newer one, as admits does (G30, answered).
  • the repo's Codex review of 3 (#25446), posted after it landed: an empty catch-up page did not join its link (G31), and a lag turned a late page of the same link into a rollback (G32). Both fixed here, with a test.

Tests

Release compile: green on the build fleet at this head (f7e8ede2e9a), fleet step eebeb251e8bbf07cb85c2c33 (scripts/cmux-next/check-release-compile.sh, Xcode 26.6), in place of the Blacksmith job.

These 9 new tests (30 in all) pass on the fleet, with fmt and clippy -D warnings clean:

  • a barred generation;
  • purges before the scan;
  • a late, older purge, before and after the scan;
  • a peer removed and added again;
  • captures that fall behind;
  • nothing queued while a loss waits;
  • flushes and losses queued once, with captures reserving their room;
  • a removal lost into a loss removes the peer whole (in "nothing queued while a loss waits");
  • past its bound, what went before the disk read takes every earlier copy;
  • a purge told again without its generation still lets the recreated session back;
  • a capture the thread never took is a loss, and a thread that ended starts again.
  • a catch-up that finds nothing new joins its link, and a lag leaves late pages late (G31, G32): with either fix reverted alone it fails, at its own assertion.

Timers, threads and loops

  • The store's thread (acpmux-peer-snapshots, Store::run), added in 2. It blocks on its channel until an op comes or the next due write or retry is reached (recv_timeout), and it ends when every sender has dropped, that is, when the hub drops. It holds no timer of its own beyond that one wait.
  • Writes go out FLUSH_AFTER (2 s) after a copy changes.
  • Retries (a write that failed, a refused removal, a root that did not read) start at 1 s and double to 60 s. Each retry stops once it succeeds, or once its copy or path is gone. A flush never cuts a retry's backoff short; only the shutdown flush tries once more.
  • flush_now (shutdown, 5) waits for its answer at most ANSWER_WITHIN (5 s), and does so through block_in_place on the multi-thread runtime.
  • No other timers, observers or polling.

Base reds, not from this PR (they fail the same way on feat-cmux-next): agent_tools::tests::own_servers_run_under_the_daemon_prefix, and the flaky session_pool::pooled_sessions_end_with_the_daemon_and_after_a_crash.

Agent session: agent session

🤖 Generated with Claude Code