#25445 acpmux peer snapshots (2): the store's thread, its bounds, writes and removals

Lawrence's list item, word for word: "start new VMs from acp chat - decouple persistence of acp trajectories from the VM itself, in case the VM is resuming".

Part of the client-snapshot piece of manaflow-ai/cmuxterm-hq#4381 ("Persistence, pause and resume, and history hydration"): the Mac's acpmux keeps a durable, bounded copy of each peer session's events, so the app paints a paused or offline peer chat at once, marked possibly stale, and catches up from the host with afterSeq. The host's journal stays the authority; the copy is not a write lease, an execution checkpoint or a backup. The shared fenced-writer database from #4381 is not part of this work.

This replaces #25323 (store) and #25378 (hook and pane), split so each piece lands on its own under about 800 changed lines, tests included. The series:

1. the on-disk form of a copy (#25448, landed)
2. the store's thread: bounds, writes, removals (#25445)
   2t. the store's tests (#25505)
3. what starts or grows a copy: generations, joins, links (#25446)
4. purges, removed peers, the queue's bound (#25447)
5. the hub's taps (wiring) (#25449)
6. the read hook _acpmux/peer_snapshot, which the VM start/resume work calls too (#25473)
7. the pane's offline banner and the reply's reading (#25474)
8. the pane paints the copy, then catches up (#25475)
   8t. the pane's tests (#25506)

This PR (2): the store's thread, its bounds, writes and removals (code; its tests are 2t)

• PeerSnapshots is the hub's handle. One std thread (Store::run) owns every copy and every file under peers/.snapshots, and the handle only sends it ops, so no file work runs on an async thread.
• Bounds.
  • Each copy is held to SESSION_EVENTS (1000) records and SESSION_BYTES (2 MiB).
  • All copies on disk are held to TOTAL_BYTES (64 MiB). Refused removals count toward it.
  • Records in memory are held to MEMORY_BYTES (16 MiB): written copies unload first, then the least recently used unwritten copy goes.
  • What each copy costs itself (KEPT_BYTES plus its key and head, loaded or not) is held to KEPT_TOTAL (4 MiB), so empty copies and copies only on disk are bounded too, the ones a scan restores included (S39, S44, S46): the least recently used written copy goes first, file and all, then an unwritten one.
  • Past the disk total, the least recently used written copy goes; an unwritten one goes only when no written one is left, and then whole, so no later record joins it with a gap (S42).
  • When a bound is passed, the least recently used copy goes.
  • A record that alone passes the session bound goes, and the copy's coverage moves past it.
• Writes. A copy is written whole, 2 s after it changes. A write that fails retries at 1 s, doubling to 60 s. A write that cannot fit under the total waits. A flush (a peer's link dropped) writes what was not tried yet and never cuts a retry's backoff short; only the shutdown flush tries everything once more (When::{Due, Pending, Last}, S43).
• Removals. A refused removal stays counted and is retried on the same backoff. When a removal empties a peer's directory, that directory goes too, by remove_dir only.
• Until the scan reads, nothing is written, and only the scan's retry wakes the thread (no spin on writes that are due). An id with no file name is dropped, never retried. A flush that could not be queued leaves no marker behind.
• The scan. Nothing is written or removed until the scan has read the root as this user's own private directory. The scan drops anything that is not a copy and cuts a copy that is past its bound.
• The thread starts at the first op; a spawn that fails installs nothing, and the next op tries again (S40). A send that fails (the thread ended) drops the sender, so the next op starts a thread again, which reads the disk first, as after a restart (G21).
• Still not wired. Its tests are the next PR (2t), split off to keep each PR under the size cap. The rules for what starts or grows a copy come in 3.

Findings from the earlier reviews this piece answers

These are S1 to S11, S16, D12 and D15 of the ledger (Codex gpt-6-astra xhigh, rounds 1 to 11 of #25323):

• metadata or segment write failures that bypassed the bounds;
• a total that left out memory and queued work;
• blocking file work on async threads;
• retry ownership;
• deletion failures, and a restart that skipped the per-session bounds;
• traffic that starved writes;
• an unbounded queue;
• one oversized record;
• an eviction clock that did not match;
• a failed scan;
• empty peer directories left behind after a scan's removals (round 11);
• this PR's fifth review: a new copy with no file taken as written (S49, written is now disk > 0 and nothing due); one large record pushing out many (S50, answered: each record goes at most once after it came, so the work is amortized O(log n) per record);
• this PR's sixth review: a thread that ended (a panic) with ops queued gave back neither their room nor their flush marks (S55): each queued op carries its cost (Queued), given back when taken or dropped untaken, and an untaken op is told to Dropped::untaken; a send that finds the thread gone starts one and sends again, once; flush_now costs nothing in the queue (W11);
• this PR's seventh review: a copy a scan restores can be its own trim's victim when written (S59); bound's + 1 saturates (S60); a head no file can hold is removed when charged (S61); the thread started after one that ended joins it and takes what it left untaken before anything else (S62).
• this PR's eighth review: charge measures the whole head line (S65); a capture while the disk does not read never brings back a write time that passed (S67); a flush mark left by a thread that ended is answered (S66: that thread's memory ended with it; the next op's thread clears the mark first).

Tests

In 2t, the next PR. This PR keeps cargo clippy -p acpmux --all-targets -D warnings and cargo fmt --check clean on the fleet, and 1's disk tests pass on it.

Timers, threads and loops

• The store's thread (acpmux-peer-snapshots, Store::run), added in 2. It blocks on its channel until an op comes or the next due write or retry is reached (recv_timeout), and it ends when every sender has dropped, that is, when the hub drops. It holds no timer of its own beyond that one wait. A thread that ended (a panic) gives back what it had queued, and the next send starts one again, once per send; the new thread first joins the one that ended (it has ended, so the join returns at once).
• Writes go out FLUSH_AFTER (2 s) after a copy changes.
• Retries (a write that failed, a refused removal, a root that did not read) start at 1 s and double to 60 s. Each retry stops once it succeeds, or once its copy or path is gone. A flush never cuts a retry's backoff short; only the shutdown flush tries once more.
• flush_now (shutdown, 5) waits for its answer at most ANSWER_WITHIN (5 s), and does so through block_in_place on the multi-thread runtime.
• No other timers, observers or polling.

Base reds, not from this PR (they fail the same way on feat-cmux-next): agent_tools::tests::own_servers_run_under_the_daemon_prefix, and the flaky session_pool::pooled_sessions_end_with_the_daemon_and_after_a_crash.

Agent session: agent session

🤖 Generated with Claude Code