cmux-tui survives a full disk: whole identity files, a journal that waits for room, units that never give up

Merged
TE
10d ago
+759-50

Leo, about this push: "Like we need to have good ass images, good ass ergonomics, good ass everything." And about the old prior art: "these are honestly old as shit. we'd better just supersede them if we have better stuff now that we know more."

Supersedes #18670 (prior art by @austinywang)

What broke

A full disk could take cmux-tui down until someone stepped in by hand:

  1. machine-id and resource-effect-pepper were created in place (create_new, then write_all). If ENOSPC hit during the write, the file was left short, and every later start failed on it as "corrupt", even after space came back.
  2. The journal writer treated SQLITE_FULL as a store that rejects the write. A batch of background events got six quick retries, then the writer stopped for good. Journaling stayed off until the daemon restarted.
  3. The systemd units that run cmux-tui had the default start limit (5 starts in 10 s). A daemon that failed on a full disk five times in a row left its unit failed, the same way acpmux went down on huge-blue (#25127).

What changes

1. Identity files are written whole or not at all (the loaders stay in workspace_registry.rs and call new helpers in workspace_registry/root_identity.rs)

  • The value is written to <name>.new (owner-only), fsynced, and renamed in, then the state root is fsynced. Any failure removes <name>.new. A stale one left by a killed writer is replaced.
  • The write still happens only once. The caller holds the file's existing lock (*.lock, flock) and has already found the file absent. Without replace, an existing file is never overwritten: the write fails and the next start reads the file that is there.
  • Cut files left by an older build:
    • machine-id: an empty file, or a strict prefix of machine_<32 hex>, is replaced with a new id. This is safe: the old code returned the id only after its write and fsync succeeded, so a cut id was never handed to any client. No registry row stores it. Any other bad text (for example machine_not-an-id) is still "corrupt".
    • resource-effect-pepper: a file shorter than 32 bytes is replaced only when no registry at schema 7 or later (RESOURCE_EFFECT_PEPPER_SCHEMA_VERSION) exists under the root. From schema 7 on, receipts are bound to the pepper, and 9-14 carry them through migration. With such a registry present, the file is kept and the start fails closed as "corrupt". The existing resource_effect_pepper_missing_corrupt_and_mismatch_fail_closed test still passes unchanged.

2. The journal writer waits out a full disk (new journal_ingress/disk_full.rs, plus a few lines in journal_ingress.rs and effect_intents.rs)

  • SQLITE_FULL, or an I/O error with ENOSPC, EDQUOT or StorageFull, no longer counts as permanent. That includes one wrapped inside another io::Error: it is checked through get_ref() as well as source().
  • Callers that wait on a receipt (barriers, producers, frontend events, registry intents) get the disk-full error right away, the same answer a store error gave them before. That covers:
    • callers in the failed batch;
    • callers already queued, drained at each try;
    • new callers, refused at admission while background events are held. So a registry caller never holds the registry lock through its 30 s deadline.
  • Events nobody waits on are kept, and the batch is retried 1 s after the first failure, doubling up to every 60 s. A refused waiter also wakes the retry, at most once a second, so once there is room the next caller finds journaling back within a second.
  • The first commit that succeeds turns journaling back on, with no restart, and logs how many events were dropped meanwhile.
  • launch_snapshot.rs has no shared helper, just inline constants. The journal uses the same 1 s doubling to 60 s schedule.
  • Memory stays bounded by dropping with a counter, on top of the existing queue bounds:
    • At most one batch's worth of events is kept (64 terminal chunks plus 256 durable events). Later ones are dropped and counted.
    • Both lanes keep their bounded queues (1024 and 256). Each drain takes at most a queue's capacity and applies the keep limit while collecting, so producers refilling a queue can't grow the drain or delay the receipts behind it.
    • A background event (terminal resize or output gap) that finds its queue full while the disk is full is dropped and counted, instead of blocking its producer until there is room.
  • Stop condition and cleanup for the backoff loop:
    • It ends at the first successful commit, which resets the delay and clears the disk-full flag. With nothing kept there is nothing to retry: the flag clears at once, and the next waiter's own commit answers it.
    • Shutdown (admission closed) wakes the wait at once through the ingress state's queue-space condvar, not a sleep. The writer then fails what it held and exits, as it does at a deadline during shutdown, so close_and_join is never held up by a 60 s wait.
    • A dropped Mux ends it at the next turn of the loop.
    • It creates no thread and no timer of its own: it runs on the existing writer thread, and an idle writer never enters it. The waiter nudge is rate-limited to once a second.
  • Not exposed in journal-writer stats: the drop counter is internal and only logged. Adding a field to JournalWriterSnapshot would regenerate every language binding, so I left it for its own PR.

3. Startup SQLite open and migration: unchanged. An SQLITE_FULL during open or migration still exits the daemon. With the units below, systemd now retries it (3 s, backing off to 60 s) instead of giving up after five tries, so I left the open path itself alone. A retry loop inside WorkspaceRegistry::open would hold the session lease and stall callers. That is not a small, safe change.

4. The cmux-tui units never give up, using #25127's settings: StartLimitIntervalSec=0 in [Unit], then RestartSec=3, RestartSteps=5 and RestartMaxDelaySec=60. The Restart= policy stays as it was (always or on-failure).

  • cmux-tui-daemon.service: web/scripts/devbox-image-common.ts devboxDaemonUnit() (the devbox bake). It was Restart=always, RestartSec=2, so five quick failures hit the default limit.
  • cmux-host.service: web/scripts/cmux-vm-image/host-agent.ts hostUnit() (the cmux-vm image bake; cmux host run supervises the session host).
  • cmux-server.service, user and system: cmux-server-core/src/units/systemd.rs, with tests/units_golden.rs updated to match.
  • cmux-host-remote.service: cmux-tui/src/remote_host/service.rs (Restart=on-failure kept, since a deliberate host serve exit stays down). The unit is now also written to a staged file beside its place, synced, renamed in, and its folder synced, so a full disk (including one a network filesystem reports only at sync) never leaves a cut unit for daemon-reload. The staged file is removed on any failure.
  • The prototype server/prototype/linux/install.sh cmux-server.service (on-failure kept). Its sleep-inhibitor unit keeps its own start limit, since it is not cmux-tui.
  • RestartSteps= and RestartMaxDelaySec= need systemd 254 or newer. The cmux-vm base is Ubuntu 24.04 (systemd 255). An older systemd logs "Unknown key name ... ignoring" and restarts every 3 s, with no limit.
  • Not changed: images/cmux-vm/ defines no unit text itself (its bake uses hostUnit). The other [Service] files in the repo do not run cmux-tui.

Review

Independent Codex (gpt-6-astra, xhigh), three rounds: round 1 FIX (pepper rule, waiter answers, wrapped errors), round 2 FIX (bounded drain, synced unit write), round 3 VERDICT: SHIP at a750ea14.

Review (codex gpt-6-astra xhigh): SHIP at a750ea140a71e3b4f220a6ea73484ce20fb1961e

Tests (fleet, Linux, 32 vCPU)

  • At head a750ea14: https://github.com/manaflow-ai/cmux/actions/runs/40266877147
    • cargo clippy ... -D warnings: clean.
    • cargo nextest run -p cmux-tui-core --lib: 2811 run, 2810 passed, 1 failed (the base flake below).
    • cargo test -p cmux-tui --lib remote_host: 16 passed, including a_failed_unit_write_keeps_the_previous_unit (a write failing partway with a full-disk error leaves the old unit and no staged file).
    • New disk_full::tests::answering_waiters_drains_a_bounded_amount_while_producers_refill: a producer refilling the queue during the drain; the drain finishes under 1 s, keeps at most the limit, and the queued barrier gets its error.
  • One commit earlier (only a clippy line changed since): https://github.com/manaflow-ai/cmux/actions/runs/40266274922, nextest 2811 of 2811 passed; crash ratchet not above base.
  • Earlier, at head 76d6bf60: https://github.com/manaflow-ai/cmux/actions/runs/40259181160
    • cargo clippy -p cmux-tui-core -p cmux-server-core -p cmux-tui --all-targets -- -D warnings: clean.
    • cargo nextest run -p cmux-tui-core --lib (the gate's runner, nextest 0.9.148, the repo's .config/nextest.toml): 2811 run, 2810 passed, 1 failed, 10 skipped. The failure is surface::tests::local_resize_order::local_resize_bounds_the_drain_to_a_queued_snapshot ("the resize consumed output without publishing its title"), a base flake: it failed in a base run (https://github.com/manaflow-ai/cmux/actions/runs/40254053930) and passed in the next one (https://github.com/manaflow-ai/cmux/actions/runs/40255362452, 2806 passed, 0 failed). This PR doesn't touch it.
    • cargo test -p cmux-server-core: 43 passed (unit goldens included). cargo test -p cmux-tui --lib remote_host: 15 passed.
  • Crash ratchet against the merge base (scripts/ci/crash_ratchet_compare.py): cmux-tui-core unwraps are 1150, the same as base. The only red row, rust acpmux: unwrap head 140, base 140, is already red on base and unchanged by this PR.
  • Under plain cargo test (all tests in one process), fork_count_bash_forks_nothing_at_startup_or_per_command_on_bash_5_3 read one extra fork in the PID gap. That test counts the system-wide PID gap, and nextest runs it alone (group fork-count-alone). It passes under nextest and alone (https://github.com/manaflow-ai/cmux/actions/runs/40254060065). This PR changes no shell integration.
  • New tests (all passing):
    • journal_ingress::disk_full::tests:
      • a_full_disk_pauses_the_journal_writer_until_there_is_room caps the registry with PRAGMA max_page_count to get a real SQLITE_FULL. The writer keeps 1 MiB of output and does not fail. A barrier sent before the writer holds the output, and one sent after, both get the disk-full error in under 1 s. Once the cap is lifted, the kept output and later output commit.
      • a_background_event_is_dropped_not_blocked_while_the_disk_is_full
      • a_full_disk_is_told_from_a_store_error (also wrapped ENOSPC, EDQUOT, StorageFull and SQLITE_FULL)
      • the_retry_schedule_runs_from_one_second_to_a_minute (1, 2, 4, ā€¦, 60, 60)
    • workspace_registry::root_identity::tests (4 tests): a cut machine-id is replaced and a foreign one is not. A cut pepper is replaced with no registry or a schema-6 one, and is kept and fails closed next to schema 7, 9 and 14. A write never replaces a file and leaves no temporary, and files are owner-only.
    • Existing tests still pass: corrupt_machine_identity_fails_closed, resource_effect_pepper_missing_corrupt_and_mismatch_fail_closed, machine_identity_files_are_owner_only, concurrent_first_open_converges_on_one_machine_identity, and the contention tests.
  • bun test tests/vm-devbox-image.test.ts tests/vm-image-devbox-metadata-guard.test.ts: 50 pass, 0 fail, including the new "the cmux-tui units never give up after repeated failed starts" (https://github.com/manaflow-ai/cmux/actions/runs/40252411839, at cfbca956; the unit text and that test are unchanged since).
  • Not run: a live VM with a filled disk.

Risk: medium. The journal writer's error path and the identity-file write run on every start, and the unit changes reach new images and cmux server install. Clients that cached the id of a never-started machine see a new one. That only happens with a cut machine-id, which the old code never handed out.

Agent session: Air-Blue

šŸ¤– Generated with Claude Code