NEXT Cloud: keep ACP trajectories outside the VM

Merged
AU
5d ago
+2938-16

Fixes #25584

Size: one feature across its two ends: the backend store and routes (~460 lines) and the VM-side mirror and restore (~1030), each with its tests (~750). Either half alone does nothing; the backend half already landing first would ship unused routes, so they stay together. It is off by default (CLOUD_TRAJECTORIES_ENABLED).

What

A Cloud machine's ACP chats (acpmux sessions) are now also kept in cmux Cloud, outside the VM:

  • Kept while the VM runs. The cmux host Cloud role on the VM (cmux-host/src/cloud/trajectory.rs, its own thread) reads acpmux's store as acpmux writes it (sessions/<id>/session.json, events/NNNNNN.ndjson). It sends every record past the server's last_seq verbatim and in seq order, and the agent's own resume file (a Claude project <sid>.jsonl, a Codex rollout) as appended byte ranges. Only complete lines are sent, every 5 s, or at once while more is waiting.
  • Readable while paused. The machine's creator reads a paused machine's chats from the API without waking the VM.
  • Restored on a replacement. A machine restored from one of its creator's snapshots reads the source machine's copy (read only). Once its acpmux runs, it imports each chat its disk lacks as an acpmux.bundle.v1 (acp session import). acpmux puts the agent's resume file back, so the history is intact and the next prompt resumes the agent. From then on the chat belongs to the new machine and is kept in that machine's copy.
  • Pause and resume. These already kept the disk and memory. The copy is the backstop when the VM is gone.

Backend (backend/apps/api):

  • Routes. POST /v1/cloud/machines/{machine}/trajectories/{append,native,list,read,native-read}, as plain HTTP outside the op path, like the keyed acpmux start.
  • Store. One TrajectoryDO (SQLite) per team and machine, in cloud-trajectory-store.ts.
  • Writes. Only the machine's own VM install can write, through the cloud.vm.* gate (vmGate).
  • Reads. That VM install, or the machine's creator (a session, or one of the creator's own installs with the read class).
  • from: "parent". A read with this field reads the restore source's store. Only the VM install can use it, and only when the person who restored the machine also created the source (MachineRow.trajectory_parent, private, set at create from a snapshot). A teammate's restore starts with no source.
  • Log rules. The log is contiguous. A batch that leaves a gap is refused with last_seq. A retried batch is a no-op. A record that differs from the kept one is a conflict, never last-write-wins. A newer createdAt (the session recreated under the same id) replaces the old copy; an older one is refused.

Defaults I picked

  • Off in production. It is off unless CLOUD_TRAJECTORIES_ENABLED=1, which is set on development and in tests only. Turning it on for staging or production is a one-line var change.
  • One store per machine, not one shared by its restores. A shared one splices two machines' logs when both run, and lets a restore from an older snapshot drop records. A chat that the snapshot's disk already has continues from the snapshot's point. Turns made after the snapshot stay readable in the source machine's copy.
  • Restore waits for acpmux to run. The importer checks the socket and never starts acpmux itself, so acpmux keeps the environment the machine starts it with. A failed import is retried, up to 3 times.
  • Files are accessed as the work user. On Linux the mirror thread uses setfsuid/setfsgid to read and write files as the session host's user, so a link the agent plants cannot make root read or write another file. Restore bundles are staged in /var/lib/cmux-trajectory-restore (owned by that user, 0700).
  • Limits. 2 GiB per machine, 1 MiB per record. A larger record stops keeping that one chat (logged), never the others.

Not covered (follow-ups)

  • cmux-harness chats. Their durable journal is not part of the bundle, so they are kept but not imported.
  • The last pass before a VM is lost. Records written in the final 5 s are not sent.
  • Deleting the copies. Copies are not deleted when their machine is. Doing that needs a retention decision, which is a new user-data delete.
  • The Mac pane. It does not read the Cloud copy yet. Leo's peer-snapshot series (#25446 to #25506) covers the Mac-side copy, and its _acpmux/peer_snapshot hook is where a server read would plug in.

Tests

  • Backend. backend/apps/api/test/cloud-trajectory.test.ts covers:
    • append, retry, gap, conflict and paging
    • a recreated session
    • resume-file ranges and path checks
    • creator-only reads while paused; only the machine writes
    • a restored machine reading its source (read only) and writing its own copy; a fresh machine having no source; the source link staying private
    • a record that names another generation
  • Rust. cmux-host/src/cloud/trajectory_tests.rs runs against a fake server with the backend's rules and covers:
    • records sent verbatim across segments with a half-written line held back
    • restart without resending; a server that lost the chat; a recreated session; disabled
    • resume files: whole lines, a rewrite, more than one pass with no new records
    • a too-large record stopping only its own chat
    • a replacement that waits for acpmux, imports the chat with its resume file, and continues it into its own copy while the source copy stays untouched
    • a deleted import that is not brought back; a failed import retried; a chat already on the snapshot disk not imported

Observed / Not run

Observed at head 743b0e8455219e211e4199927620542bd5dba8f5:

  • Backend typecheck and all 1,367 API tests across 225 files passed, including all 11 trajectory tests: https://github.com/manaflow-ai/cmux/actions/runs/41157308621/job/124406826230.
  • On rbx, cargo test --locked -p cmux-host cloud::trajectory_tests -- --nocapture passed 18 tests. This includes the 501-session restore and five source-change boundaries.
  • The focused regression command cargo test --locked -p cmux-host cloud::trajectory_tests::a_restore_rejects_generation_changes_and_relists_before_retrying -- --nocapture failed at cab3d3f54fd0657c9968647c8038070305b43ea1 for mixed session generations and at 951627f212ae9aa0b3f7a685d06f7ae459f8a428 for a native-file rewrite within the same generation. It passes at this head. Each failing regression was committed before its repair.
  • On rbx, cargo clippy --locked --no-deps -p cmux-host --lib --bins -- -D clippy::indexing_slicing passed. The static pre-push gate also passed without bypasses.
  • Fleet Release check: cmux-ci compile-check --ref 743b0e8455219e211e4199927620542bd5dba8f5 --app --release --recipe-version 2 exited 0, job 7b3748cbbfb0fd862e2d1d8a. The app and bundled cmux-tui receipts are both ok: true on this exact head. The earlier fleet app check also passed (job 862e93fba177df25ffcf4d69).
  • Two independent correctness and impact reviewers returned SHIP for this exact head: https://github.com/manaflow-ai/cmux/pull/25682#issuecomment-6537609413.

Inherited checks, with matching base evidence:

Landing infrastructure: cmux-next/compile timed out with no result from its results endpoint despite the successful fleet Release receipt. Autoland still attributes the inherited guards failure to this PR. Exact-head receipts and matching base logs are posted at https://github.com/manaflow-ai/cmux/pull/25682#issuecomment-6537873443.

Not run: a live VM-loss/replacement flow or actual acpmux import. The remote tests use a fake server with the backend's rules. No Swift or visible UI changed in these repairs. The feature remains off in production. The earlier tagged app is historical evidence, not validation of this head.

Changelog

none (off by default; no user-visible change until it is turned on)

šŸ¤– Generated with Claude Code

2026-10-29: complete trajectory pagination (HQ #5074 review follow-up)

Fixed the 500-session restore cutoff identified in review 5702106134:

  • The TrajectoryDO store now lists sessions by immutable session ID in pages of at most 500, returning has_more and next_after_session. Mutable updated_at is no longer the paging key, preventing missing/duplicated rows during concurrent metadata updates.
  • /trajectories/list validates the optional after_session cursor and preserves the same completion fields for empty parent stores.
  • The VM-side Rust Mirror::prime requests every page before declaring import ready. Missing/unchanged cursors, malformed rows, unordered pages, and old backend replies containing a full 500 rows without a continuation flag fail visibly instead of claiming completeness.
  • Added a backend test with 501 persisted chats and a Rust fake-server test which primes/imports all 501 across two pages, including chat-500. A legacy 500-row reply without a cursor is rejected.

The 501-session backend and Rust regressions pass at the current head; see Observed / Not run above.

Rolling-version safety: When the store has 501+ chats, a VM guest that lacks pagination support receives 409 cloud.trajectory.pagination_required instead of a plausible truncated 500-row list. Updated guests send page_protocol:2 on each read and follow the cursor. Short legacy listings remain parseable, including their historical newest-first ordering; exactly 500 without a continuation flag is treated as ambiguous and refused. Native-file restores require the revised backend: missing generation or revision fences fail closed. Added a legacy-short-list compatibility test in Rust and an old-client 501-chat HTTP refusal test.

Restore consistency

List pages consume the SQL cursor only through the 4 MiB byte budget plus one lookahead row. Every event page checks the listed session generation; every native-file chunk checks both that generation and the listed file revision. A rewrite changes the revision, while appending preserves it. A source change discards staging and refreshes the list before retrying, so a restored bundle cannot mix old metadata with a new session or native file. Existing files acquire stable revisions when read.