acpmux: keep a durable, bounded copy of each peer session's events (client snapshot, hq#4381)

Closed
TE
10d ago
+2293-19

Lawrence's list item, word for word: "start new VMs from acp chat - decouple persistence of acp trajectories from the VM itself, in case the VM is resuming".

This is the daemon store for the client-snapshot piece of manaflow-ai/cmuxterm-hq#4381 (docs/remote-acpmux.md, "Persistence, pause and resume, and history hydration"). The Mac's acpmux now keeps a durable, bounded copy of each peer session's events. The app can then paint a paused or offline peer chat at once, marked possibly stale, and catch up from the host with afterSeq. The host's journal and native files stay the authority. The copy is not a write lease, an execution checkpoint or a backup. The shared fenced-writer database from #4381 is not part of this PR.

The read hook, _acpmux/peer_snapshot {sessionId, host?, limit?, maxBytes?}, and the app's paint-then-catch-up follow in the stacked PR. The VM start and resume work is meant to call the same hook.

What it does

  • What is captured, with no behaviour change:

    • _acpmux/attach and _acpmux/events pages this daemon forwarded for a peer's session (server/peer_forward.rs, after strip_asks), only for a session the peer still lists;
    • the peer's live records (hub/peers.rs apply_peer_notice, after the ask filter).

  • On disk (hub/peer_snapshot/disk.rs), beside peers/<peer>.json, each session gets peers/<peer>/<muxSessionId>/. It holds:

    • meta.json: generation, lastSeq, capturedAt, and the last-known row without its retry countdown;
    • seg-<n>.ndjson segments.

  • How segments are written:

    • A segment is always written whole, from its parsed records plus only the seqs it does not hold yet, to a temporary file that is then renamed (config::write_atomic). A full disk or a crash never leaves a cut segment, and a retry adds nothing twice.
    • A segment with a line that does not parse is never written again.
    • A record that does not fit the open segment starts the next one.

  • What a start reads back: it follows no symlink, and treats a session directory without meta.json as no copy and removes it.

  • Bounds:

    • 1000 records and 2 MiB per session; past that, the segment holding the oldest records goes, after a meta.json that no longer counts it is written.
    • 64 MiB in total, on disk and in memory (records, meta.json, row); past that, the least recently captured session goes.
    • Unwritten records: one session's bound each and 16 MiB in all.

  • Dedupe: by seq, against both disk and memory.

  • When the copy is dropped instead of merged:

    • the session's createdAt changes (the session was recreated);
    • a page's lastSeq is below the copy's on any link but the one that last paged it (a VM came back on an older disk, or this daemon restarted);
    • the session is purged on the peer, purged through this daemon, or its peer is removed. A purged session is not captured again in this run.

    A live record on a link no page has checked yet is not kept.

  • Removal: meta.json goes first. A removal the filesystem refuses is logged and tried again before the next write, and no copy is written under that path until it is gone.

  • Write failures: a failure such as ENOSPC is logged and the records stay in memory within the bounds above. The write is tried again at 1 s, doubling to 60 s, as daemon/persist_retry.rs does. It never exits.

  • Blocking IO: the background write runs on spawn_blocking. Inline file work from async callers goes through block_in_place on the daemon's multi-thread runtime, as persist_retry.rs and session_scope.rs do.

New timers and loops

  • Per-session write task (PeerSnapshots::schedule): a one-shot tokio::spawn that sleeps 2 s and then writes once on a blocking thread.
    • Stop condition: that one write. It schedules exactly one follow-up only if the write failed (backoff 1 s to 60 s), the batch filled a segment, or something changed meanwhile.
    • A session has one task at most: each schedule takes a new ticket, and a task that wakes holding an older ticket ends.
    • If the session's copy was dropped meanwhile, the task ends.
    • An idle daemon starts none.
    • Cleanup: Hub::shutdown_all writes what is left, and the runtime ends any sleeping task with the daemon. A peer going down also writes its sessions at once.
    • At shutdown a refused write may lose the last seconds of a copy, never the chat.
  • No other timers, observers or loops.

Size

About 1400 changed lines, a third of them tests. That is over the usual 800. The store cannot land in halves that each do something: the capture, the bounded write, and the drop rules only make sense together. Any smaller cut would leave unused code behind allow(dead_code).

Coordination

hb-acpmux (the acpmux daemon owner) confirmed there is no conflict. It expects a one-line rebase in the initialize methods list for the hook PR. The retry row field from its retry work is stripped from the persisted summary, and a test covers that.

Review

Codex gpt-6-astra at xhigh, on cmux16, over three rounds. The first two returned FIX. Every finding was addressed:

  • the purge race, restart rollback, a torn-line append, read errors, batch bounds, the total bound, blocking IO, and retry ownership;
  • an idempotent retry, segment byte bounds, link checks, metadata durability, removal retries, and symlinks.

The verdict at this head is in the comment below.

Tests

On the fleet (Linux, 32 vCPU), cargo test -p acpmux --lib passes (850), and cargo fmt --check and cargo clippy --all-targets -D warnings are clean.

hub/peer_snapshot_tests.rs covers:

  • surviving a restart, with dedupe by seq, where a live record on an unchecked link is not kept;
  • the per-session bound;
  • older history paged in later being evicted before the newest records;
  • the total bound with least-recently-captured eviction;
  • a full disk (ENOSPC), with a background retry until the write lands;
  • a retry after a failed meta.json write adding nothing twice;
  • a record too big for the open segment;
  • a recreated or restored session, including after a live record on the new link;
  • an empty page leaving no copy;
  • a down peer's records and forwarded pages, with no retry countdown, and purge and peer removal.

hub/peer_snapshot/disk_tests.rs covers:

  • a cut line, a leftover temp file, and a session without meta.json;
  • symlinks;
  • idempotent segment writes;
  • removal order and the scan bound.

tests/daemon_chat_open.rs (an open-bytes budget) fails identically on base (5.6 MB against a 4 MB budget), so that failure is not from this PR.

Risk: low. Only peers' sessions are touched, reads are unaffected, and capture never fails a request.

Agent session: agent session

šŸ¤– Generated with Claude Code