agent pane peer snapshots (8): a peer chat paints the Mac's copy at once, then catches up

Merged
TE
9d ago
+513-15

Lawrence's list item, word for word: "start new VMs from acp chat - decouple persistence of acp trajectories from the VM itself, in case the VM is resuming".

Part of the client-snapshot piece of manaflow-ai/cmuxterm-hq#4381 ("Persistence, pause and resume, and history hydration"): the Mac's acpmux keeps a durable, bounded copy of each peer session's events, so the app paints a paused or offline peer chat at once, marked possibly stale, and catches up from the host with afterSeq. The host's journal stays the authority; the copy is not a write lease, an execution checkpoint or a backup. The shared fenced-writer database from #4381 is not part of this work.

This replaces #25323 (store) and #25378 (hook and pane), split so each piece lands on its own under about 800 changed lines, tests included. The series:

  1. the on-disk form of a copy (#25448, landed)
  2. the store's thread: bounds, writes, removals (#25445)
    2t. the store's tests (#25505)
  3. what starts or grows a copy: generations, joins, links (#25446)
  4. purges, removed peers, the queue's bound (#25447)
  5. the hub's taps (wiring) (#25449)
  6. the read hook _acpmux/peer_snapshot, which the VM start/resume work calls too (#25473)
  7. the pane's offline banner and the reply's reading (#25474)
  8. the pane paints the copy, then catches up (#25475)
    8t. the pane's tests (#25506)
    8t2. tests of 8's attach fixes (#25620)

This PR (8): a peer chat paints the Mac's copy at once, then catches up (code; its tests are 8t)

This is the pane side of the stop condition: a paused VM's chat shows its history at once.

  • Open. A chat on another machine (a Cloud VM, huge-blue) that this page holds no tail of opens on the Mac's copy (drawPeerCopy), drawn as a kept tail. The pane waits at most PEER_COPY_READ_MS (400 ms) for it. A chat on this Mac never asks for a copy.
  • Offline. While the machine does not answer (peer_unreachable), the copy stays, marked offline under the banner from 7 (keepPeerCopy), and the page stays open. A chat this page read before whose machine is now offline shows the daemon's copy in place of its cached tail.
  • Catch-up. Once the row says the machine is back (peerBack), the chat attaches from the copy's newest seq, as a kept tail:
    • a page that joins the copy continues it, each seq once;
    • a page that does not join, a recreated session (another createdAt) or a machine back on an older log replaces the copy;
    • in every case, the live records that came while the attach was out are kept, each seq once (copyLive, deduped by seq: P12).
  • Session check (P11). A copy is kept only while it is of the session listed now. If the session was recreated while the copy's attach was out and the machine is down, the old copy goes and the copy of the current session is read if the Mac has one. A row naming another session while the machine stays down also takes the old copy off screen.
  • Recreated while offline. A row naming a new session while the machine stays down draws the Mac's copy of the new session, offline, if it has one (redrawPeerCopy), and reattaches it when the machine is back. The redraw starts from the same session reset a switch uses, so the old session's queue and unsent prompts never show under the new one (P19).
  • Which copy (P18). A reply is drawn only if it names this session (sessionId or qualifiedSessionId), and a copy stands only while its row names the same machine and generation (copyOfRow). A row that moves the session to another machine takes the old machine's copy off screen and draws that machine's copy, if the Mac has one.
  • Machine moves (P20). An attach over a copy of another machine than the row names now replaces it, so two machines' rows never join.
  • Back during an attach (P17, P21). A row that says the machine is back while the copy's attach (or a redraw's read) is out is remembered (copyBack); if that attach ends offline, the copy reattaches once. Every read of the copy is marked (drawPeerCopy), the fallback read after a failed attach too, and an attach that succeeds clears the mark. Each such row change causes at most one more attach, never a loop.
  • Another identity after a failed attach (P22) resets the session before the new copy is drawn, as redrawPeerCopy does.
  • Rewind check. A machine back on an older log is told by the reply's pageThrough (5), falling back to lastSeq for older peers.
  • Failure. Any other failure takes the copy off screen and forgets the tail, as a failed open does.
  • Swift. AgentPaneTailCache (Swift) is unchanged by design: the daemon's copy is the durable source, and the webview's tail cache stays a fast path.

Capture

The pane state this PR draws, rendered from the gallery on the fleet:

Before (the chat had nothing to draw)After (the Mac's copy, at once, marked possibly out of date)
Dark
Light

The after state is what 8 draws when a paused or offline VM's chat opens: its history shows at once, its header names the machine (dev-vm-2, a fixture name), and the banner says it is the Mac's copy.

Real app (peer taken down on a fleet slot)

The real app, built at 77ffe44cb9 (feat-cmux-next with cmux#26206), running in a fleet capture slot. This is a peer taken down, not a paused Cloud VM:

  1. The app's acpmux has an ssh peer daemon (devbox) that holds a chat with history.
  2. The chat is opened while the peer is up.
  3. The peer daemon is shut down.
  4. The chat is opened again.

The attach fails with peer_unreachable, and the pane paints the history from the Mac's copy under the offline banner (connection peer offline, 11 rows). Before cmux#26206, the relay refused _acpmux/peer_snapshot, and this same rig showed an empty Failed chat.

Findings from the earlier reviews this piece answers

These are P1 to P12 of the ledger (Codex gpt-6-astra xhigh, rounds 1 to 6 of #25378):

  • a cached tail losing its generation;
  • a lost gap cursor;
  • live records lost on recreation;
  • failed recovery leaving a hole;
  • a stale copy that was never marked;
  • a late host-tail reply;
  • a VM back on an older log;
  • a reconnect bypassing the copy;
  • concurrent attaches;
  • an offline chat that closed;
  • a copy of an old generation kept offline (P11, fixed here);
  • duplicate live records drawn twice (P12, fixed here);
  • this PR's second review: back ignored while attaching (P17), copy identity by generation only (P18), a redraw that kept the old queue (P19);
  • this PR's third review: a machine move during an attach (P20), the fallback read losing a back row (P21), a replacement after a failed attach keeping old state (P22);
  • this PR's fourth review: the fallback resets the session when the rows on screen are of another createdAt than the row names, held copy or not (P24, rowsGeneration); a row change while the copy is read reads it once more (P25, one retry only);
  • this PR's fifth review: a reconnect owns the copy's attach from its start (P27); a re-read copy of another session resets the session before it is drawn (P28); each read has its own change marker (P29);
  • this PR's sixth review: an attach that finds a recreated session resets the old one's state (P30); the rows' machine is tracked too (rowsPeer, P31); a recovery attach that a reconnect took over leaves the copy to it (copyAttaches, P32).
  • this PR's seventh review: rows of a peer chat not proven of the row's machine are reset before the page, and kept tails record their machine (P33); a row saying the machine is back while an attach with no copy drawn is out is read once after it fails (attachOut, P34); dropping the copy resets the session's cursors (P35); held live records are one per seq (P36).
  • this PR's eighth review: a copy read of a selection that moved on never starts (P37); a tail of a machine never joins a chat on this Mac (P38).
  • this PR's ninth review: an attach reply of the session before a recreation is dropped and the session listed now attached, once (P39); every attach of the chat takes a token that a reconnect moves on, and the redraw's attach is owned and its failure handled (P40); qualifiedSessionId is answered (P41: the pane holds the qualified id).
  • this PR's tenth review: a reply after a move to another machine is attached again from that one (P42); a recreated session that is back drops the old copy before it is read (P43); the watermark and drewKeptTail findings are answered (P44, P45).
  • this PR's eleventh review: a late record of an attach that failed is not shown above the offline copy (P46); every attach reply is held to the row as it answers, a retry's too, up to ATTACH_TRIES (P47); a row that changed while the attach was out to any other createdAt is another session, an earlier one too (P48).
  • this PR's twelfth review: every attach changes nothing once a later one moved the token on, on success as on failure (P49); a row that named another session while the attach was out drops what is drawn and attaches again (P50); an open that fails before its attach leaves the copy offline (P51).
  • this PR's thirteenth review: the copy's read answers how far it covers, not its newest record (P52); the row is compared with the session asked of too (P53), and watched until the page is drawn, the reads after the reply included (P54).
  • this PR's fourteenth review: a kept tail names only the machine and session its rows are proven of, never what a later row put in the summary (P55); the cursor after a copy is trimmed past a refused attach's record is the newest record kept, as rebuild sets it (P56).
  • this PR's fifteenth review: an attach that fails with no copy to draw reads the machine once more when its row said it was back meanwhile (P60); one that fails after its row named another session drops what it drew before the copy is read (P61). Its other three points are answered in the ledger without a change (P57 to P59: records past the copy are read again from the machine; a tail with no peer is a local chat's in this build; a cold attach's failure keeping streamed rows is the same on feat-cmux-next for every chat: filed as #25967).
  • this PR's sixteenth review: the attach of a chat already left (its select waited on a detach that answered late) returns before it watches the row, so it never takes the watch of the attach out (P62).
  • this PR's seventeenth review: a chat read ahead keeps nothing when its row named another session or machine by the time the page landed (P64). Its other two points (an attached chat's row changing with nothing out, #25968, fixed in #25973; and a reconnect's attach beside a select's, #25969) are how feat-cmux-next behaves for every chat (P63, P65).

Tests

In 8t, the next PR, split off to keep each PR under the size cap. This PR keeps the webview typecheck clean on the fleet.

Timers, threads and loops

  • PEER_COPY_READ_MS (400 ms) is one setTimeout per open, raced against the read. It resolves once and holds nothing.
  • copyLive is held only while an attach over drawn rows is out. It is cleared when the attach answers, fails, or the chat changes.
  • copyBack is one field, set by a row change while an attach or read is out and cleared when it is used or a reattach starts. It holds no timer.
  • attachWatch holds the sessions and machines the row named while one attach ran (from its ask until its page is drawn), one entry each; it goes when that attach returns (finally). No timer.
  • ATTACH_TRIES (3): an attach whose reply is of another session or machine than the row names when it answers (the row changed while it was out) is asked again, at most 3 times in all; past that the open fails. No timer, so it is never a loop.
  • peerCopyReading is one object per read ({ sessionId, changed }), set when the read starts and cleared by that read only when it settles. A read whose row changed meanwhile and whose reply is rejected is redone once (again is false on the redo), so it is never a loop.
  • No polling. Catch-up starts from the row's session_changed, once per offline copy (attaching).

Risk: medium, limited to peer chats. A chat on this Mac takes none of the new paths.

Base reds, not from this PR (they fail the same way on feat-cmux-next): bun run check exits 1 there too, and so does gallery-coverage (it lists components that are not from this PR).

Agent session: agent session

šŸ¤– Generated with Claude Code