Fix cmux-tui VM state and journal reliability

AU
1mo ago
+1152-61

Summary

Cloud VMs could lose their cmux-tui socket after the state filesystem filled: SQLite/WAL writes failed, journal maintenance could not make progress, and checkpoint/list requests could amplify the failure during reconnect and startup. This change keeps the state filesystem reserved, grows it safely when possible, bounds journal catalog responses, and retries transient checkpoint capture races.

  • Grow the reserved ext4 state image online in bounded steps up to 8 GiB, preserving a 1 GiB root-filesystem reserve; resume partial resizes and serialize boot/upgrade growth with a state lock; defer daemon startup when growth is unsafe or unavailable.
  • Ship the current state-mount helper during the running-guest upgrade path, including crash-looping/no-daemon guests, and restore it alongside a binary rollback.
  • Bound checkpoint and segment list responses to 4,096 entries, matching the embedded resource-operation result contract.
  • Retry bounded transient checkpoint capture/commit races instead of treating concurrent journal activity as a terminal reconnect failure.
  • Promote and verify a fresh 12-entry Freestyle desktop/base ladder from this exact source, so new machines carry the state reservation and growth helper.
  • Document the storage and rollout contracts and add a regression test for bounded journal lists.

Testing

  • bun test web/tests/vm-devbox-image.test.ts --test-name-pattern 'reserves an isolated|every shell file parses' (2 passed)
  • bun test web/tests/vm-image-manifest.test.ts web/tests/vm-devbox-image.test.ts --test-name-pattern 'checked-in image manifest|reserves an isolated|every shell file parses' (6 passed)
  • bun run devbox:manifest:check (passed: 12 validated Freestyle defaults, current source digest matches)
  • Hosted Rust verification workflow: 37850659849 (all Rust packages, MSRV, consumers, and seven-language live conformance passed on the final head).
  • Fresh Freestyle bake and verification completed from source head 986c868fe2e906bcea5c7820ea2d573e0c5718ee; all image checks passed and six derived sizes were booted and checked.

Impact map

  • Source of truth and owner: web/services/vms/images/devbox/cmux-tui-state-mount owns the reserved state filesystem; journal_extensions.rs owns journal catalog bounds; mux.rs owns checkpoint orchestration; manifest.json records the verified image ladder.
  • Direct callers and sibling entrypoints inspected: image bake and boot supervisor, fleet upgrade runner and guest rollback script, reconnect checkpoint callers, and journal list protocol handlers.
  • Cross-surface effects: new guests get bounded state growth; existing guests receive the helper through the upgrade script; no VM size or provider changes.
  • Tests and evidence: focused image-template tests, manifest contract check, hosted Rust/Linux verification, and fresh Freestyle image verification.
  • Residual risk: the state image remains capped at 8 GiB and keeps a root reserve; a machine whose journal exceeds available disk still defers startup and retains its canonical history for operator recovery.

Mergeability

  • Base SHA: 5244a7555c5ae8d528127af5a1946f5c8de99c5e
  • Head SHA: acc70de861c396c22d4696e4fd88ff4769b97062
  • Conflict-only gate: PASS (62b22154d6789d11acf47083003f2ad817ea96b9)

[!NOTE]
Medium Risk
Touches guest disk growth, fleet upgrade/rollback, and journal checkpoint semantics on running VMs; bounded retries and caps reduce blast radius but state growth still depends on root free space and the 8 GiB cap.

Overview
Improves cmux-tui durability on cloud VMs when the reserved state disk fills or journals race during reconnect: the state-mount helper can grow the ext4 image online (double steps, 8 GiB cap, 1 GiB root reserve, flock + resize retry safety), fleet upgrades install that helper before touching the daemon (including crash-loop / no-daemon cases) and restore it on binary rollback.

On the daemon side, journal checkpoint create now retries up to four times on transient capture/commit races (session/terminal/journal movement during capture), and checkpoint/segment list queries are capped at 4,096 rows with spec/tests/docs aligned.

Rollout: upgrade-fleet-cmux-tui bundles the current cmux-tui-state-mount; manifest promotes a new Freestyle state-reliability image ladder on cmux-tui 11bd707349 and demotes the prior defaults.

Reviewed by Cursor Bugbot for commit 986c868fe2e906bcea5c7820ea2d573e0c5718ee. Bugbot is set up for automated code reviews on this repo. Configure here.