cmux-journal: one append-only journal per daemon, with group commit and ack after durable (R-DURABILITY S1)

Merged
LA
2d ago
+5168-0

Summary

Stage 1 of the R-DURABILITY rewrite (design: handoff/rewrites/R-DURABILITY/DESIGN.md, STATUS 9088). Today each daemon has many durable write paths (three SQLite databases, per-terminal tail and snapshot files, per-chat logs, session.json, about 30 small JSON files), each with its own writer, order, sync rule and recovery. That split is the common cause of REOPEN-F3 (#31643 and its 7 review rounds), acpmux g2 (#29311, #30120), the session.json vs log split, and the acked-but-unsynced changes the design lists (E9, E12-E15). The rewrite moves every acknowledged change of each daemon into ONE append-only journal with group commit, and a reply goes out only once its record is synced.

This PR adds only the journal crate, cmux-tui/crates/cmux-journal. Nothing uses it yet; the next stages wire it into acpmux (event logs, then chat metadata) and the cmux-tui daemon, each behind one switch.

  • Journal::append copies one record into memory and returns its LSN: no I/O, no wait. One writer thread takes every pending record as one batch, writes it with one write, and syncs it once when it holds an acked record (group commit). Durable listeners then release the replies that wait for those LSNs. Stream records (terminal output, streamed tokens) are written in the next batch and synced with the next acked batch, or 1 s later.
  • Sync levels: F_BARRIERFSYNC per acked batch on macOS (fdatasync on Linux). The first barrier sync starts a 1 s window; then one batch (or one sync alone, when no batch comes) uses F_FULLFSYNC. So a power loss takes back at most 1 s of acked batches, always a suffix. On macOS a failed F_BARRIERFSYNC / F_FULLFSYNC falls back to fsync only when the volume does not support the call (ENOTSUP, EOPNOTSUPP, EINVAL, ENOTTY); a call that a signal interrupts (EINTR) is made again; any other error is a failed sync.
  • Sync markers: after each successful sync that covered records, the writer appends a marker entry (no payload) that vouches for them; a seal and a rotation sync the last marker too.
  • Recovery is the one start path (clean exit, crash, kill -9, OS crash). It fully syncs the last segment, checks every entry's CRC and LSN, and hands the records after the caller's compaction point to the caller's replay. It trusts the last segment only up to its last marker: the records after it may sit in pages that a failed sync dropped, so recovery writes them again into a new segment (written and fully synced as <name>.seg.new, then renamed, folder synced) before open returns. The journal always goes on in a segment this open made. A torn tail stays in its file. Damage that is not a torn tail moves the later segments to quarantine/ (never deleted). No LSN is ever used twice: every LSN of every segment, live or quarantined, counts at every open; a header that does not read is read for its LSNs from its first record.
  • A failed write is cut back to its size before the write and written again in the same segment. A failed sync is never retried on that segment (after a failed fsync the pages can be clean and lost): every record after the durable LSN goes into a new segment. A panic on the writer thread closes the journal: appends fail (Closed), waits end (WriterEnded), listeners are dropped.
  • Storage is the file system seam. The sim feature's SimStorage simulates a page cache over a disk (4 KiB pages), Linux write-back errors (a failed sync drops pages that a later sync never writes), one ordered log of names, process crashes, OS crashes and power losses, also inside recovery. Later stages use it for store-level crash tests of the daemons.

OWNERS.md gets a row for the crate.

Review fixes (independent review FIX at 89293fb15f39, comment 6610468202)

findingfixregression test (red on the code before the fix)
F1 a crash inside quarantine reuses quarantined LSNsquarantined LSNs count at every open; an unfinished quarantine is reported againa_crash_inside_the_quarantine_step_never_reuses_a_quarantined_lsn (every operation of the open is a crash point once)
F2 a damaged last header reuses its LSNsthe segment is read for its LSNs from its first record; the bytes after a cut are scanned tooa_damaged_header_in_the_last_segment_never_reuses_its_lsns
F3 a listener panic hangs waitersa guard closes the journal when the writer thread endsa_panic_on_the_writer_thread_closes_the_journal
F4 a restart after a failed sync trusts the failed segmentsync markers; recovery writes the unvouched records again; never goes on in an old segmenta_restart_after_a_failed_sync_keeps_every_later_ack, and in the crash sim a_kill_after_a_failed_sync_then_os_crashes_keep_every_acked_record
F5 the sim could not find F1 or F4SimStorage rewritten (page cache, write-back errors, partial failed writes, ordered names, crash inside recovery, aimed faults)4 sim self-tests
F6 one fd per segment at open; no retirementone file open at a time; retirement is an S2 API (before the first consumer)recovery_holds_at_most_two_files_open
F7 two docs contradict the codeformat.rs and lib.rs corrected

Also: the EINTR retry (a93684835c4).

Re-review fixes (FIX at 3498975b5267, comment 6620113993)

findingfixregression test (red on the code before the fix)
G1 the copy re-read the file, unchecked; an evicted dropped page gave zeros under a vouching markerrecovery stages the bytes its checked read returnedrecovery_copies_the_records_it_checked_when_the_cache_drops_them (the sim now evicts dropped pages after a read)
G2 a crash at the open's last step lost the damage reportthe damage goes to DAMAGE (whole, synced) before the new segment hides it; every open reports it until Journal::clear_damagethe F1 regression asserts the report (lost_through 40) at every crash point
G3 a symlinked state folder failed every openfolder syncs follow symlinks (no O_NOFOLLOW)a_journal_in_a_symlinked_folder_opens_and_round_trips (real file system)
G4 every open read every quarantined segmentthe quarantine is read only while no live segment is numbered above itan_open_after_a_finished_quarantine_reads_no_quarantined_segment
G5 sim.rs over the function limitsplit: sim/machine.rs (the machine model)
G6 one tracked arithmetic hitchecked_div

Third-review fixes (FIX at cb893c6b64cd, comment 6621681018)

findingfixregression test (red on the code before the fix)
G7 the quarantine scan was skipped behind a newest segment whose header is damaged; a header-less quarantined file was scanned from too low a baseskip only after a segment whose header checked out is numbered above the quarantine; scans start from the running floora_damaged_header_after_a_quarantine_never_reuses_a_quarantined_lsn (A), a_damaged_newest_segment_after_a_quarantine_reports_its_records (B)
N1 one loss reported twice (after a crash, or by two steps of one open)a loss inside another is recorded and reported once, as the larger rangethe F1 regression asserts that no reported range lies inside another
N2 docsDAMAGE named; long doc lines wrapped
recovery.rs near its limitsplit: recovery/damage.rs, recovery/scan.rs

Fourth-review fixes (FIX at 215c8f0485ea, comment 6622740919)

findingfixregression test (red on the code before the fix)
G8 a recovery copy is numbered above the quarantine but its records start below the quarantined LSNs; with the empty segment after it damaged, nothing that is read carried the flooreach quarantined file is named <segment>.<n>.<floor>; every open takes the quarantine's floor from the names alone (no file read, no header trusted)a_recovery_copy_above_a_quarantine_never_reuses_a_quarantined_lsn (the reviewer's three steps)
N3 an absorbed loss dropped its files' namesthe kept loss takes the absorbed one's files and reasonG7 B and the F1 test assert that every quarantined file is named by a reported loss
N4 DAMAGE was written after the movesthe quarantine is planned (measured and named), the loss recorded, then the files movethe F1 test asserts, at every crash point, that the loss through 40 names the damaged segment

Fifth-review fixes (FIX at 9c47594e1c04, comment 6623794646)

findingfixregression test (red on the code before the fix)
G9 an open that gives out LSNs above everything its kept and new segments say (damaged synced records or torn records in the last segment, which the recovery copy then hides from every later open) kept that floor only in the header of the empty segment it made; with that header damaged, the next open gave those LSNs out again (clear_damage also removes the DAMAGE line, and torn records are not damage at all)the open makes the empty file FLOOR.<lsn> and syncs the folder before it makes a segment; every open takes the highest floor from the folder listing (no file read), as it does for the quarantine names. The caller's compaction point, the quarantine names and this open's moves carry their floors already: no file is made for thema_skip_hidden_behind_a_recovery_copy_is_never_given_out_again, torn_lsns_hidden_behind_a_recovery_copy_are_never_given_out_again

Testing

Two test files (crate cmux-journal):

  • src/cmux_journal_tests.rs (filter cmux_journal_tests, 35 tests): the record and segment format; group commit (400 acked records from 8 clients with a 20 ms sync take fewer than 200 syncs); an ack is not durable before its sync ends; listeners in order; stream records; an idle journal does no storage operation; the full-sync window; backlog and size limits; seal; the lock; rotation; failed writes and syncs; a failure at any of 11 operations keeps every acked record; torn tails; quarantine; compaction points; replay errors; the review regressions above; a round trip on the real file system, also in a symlinked folder.
  • src/crash_sim_tests.rs (filter crash_sim_tests, 9 tests): 200 seeds each of process crashes, OS crashes, power losses and a mix, over several epochs per seed (appends, a crash at a random operation or after an aimed failed sync, an open that may crash inside recovery). After each open: every record byte for byte, every record an earlier open replayed still there, the epoch's records a prefix of its appends, every acked record kept (fully durable after a power loss), no damage. One run forces the F4 order on every seed. 4 tests check the simulated machine. CMUX_JOURNAL_CRASH_SEEDS=N runs more; CMUX_JOURNAL_CRASH_SEED=S reruns one seed.

Red on the code before the fix (the tests' commits on top of 89293fb15f39):

  • cmux_journal_tests at 263da86f2235: fleet c4178647 (hq run 41720702331): 22 passed, 5 failed (the 5 review regressions), each also failed alone.

  • crash_sim_tests at 4dfdffc0d58a: fleet ccb22f4d (hq run 41746999434): a_kill_after_a_failed_sync_then_os_crashes_keep_every_acked_record failed 200 of 200 seeds ("records are missing before the next segment" after the OS crash), and the mixed run failed 5 seeds; both failed again alone.

  • Re-review regressions at 644e77a4c52e: fleet de35f6d0 (hq run 41759247817): 26 passed, 4 failed (G1: [1], wanted [1, 2, 3], with "a synced record is damaged"; G2: damage: [] after a crash at op 69; G3: "sync the parent folder: Not a directory"; G4: a quarantined segment was read), each also failed alone.

  • Third-review regressions at 936b8ace4288: fleet 27d011ca (hq run 41767569544): 29 passed, 3 failed (G7 A: next append Ok(15), wanted Ok(41); G7 B: "the loss of 41..43 is not reported"; the F1 test's N1 check at op 64), each also failed alone.

  • Fourth-review regressions at 4da46785b394: fleet 0166dc1b (owned Mac aws-m4pro-4; the Blacksmith run 79bcd79e died on a full runner disk before its tests): 30 passed, 3 failed (G8: Ok(6), wanted Ok(11); the F1 test and G7 B: a quarantined file not named), each also failed alone.

  • Fifth-review regressions at 65cead07fed0: fleet 3372e3af (owned Mac aws-m4pro-8): 33 passed, 2 failed (G9 skip: the next append got Ok(5), wanted Ok(12); G9 torn: Ok(4), wanted Ok(6): LSNs given out again), each also failed alone.

Green at the head 51cfdee876b2a4dafe4de970e211be9d130e3e26:

Observed / Not run

  • Observed: the fleet steps above (Linux: Blacksmith 8-vCPU Ubuntu 24.04; macOS: owned Macs aws-m4pro-*).
  • Found on the way: the first crash-sim red step (fleet 184129d8 at 263da86f2235) passed on the old code. Its random faults reached the F4 order in 1 of 800 runs, and that run passed when it ran alone. The forced run and the aimed faults (commit 4dfdffc0d58a) make it fail every time on the old code.
  • macOS: the crate's tests ran on owned Macs (above), the real-file-system tests included; the crash sim last ran on a Mac at b589044b852 (G9 does not change the simulator or the crash runs). The 100,000-seed acceptance run (DESIGN.md A2) is a fleet measurement, not part of this PR.

Changelog

none

Proof

No capture: a Rust crate with no user-visible change and no consumer yet.