PR #15153 - manaflow-ai/cmux

Problem

A production Cloud machine's cmux-tui registry was 12 GB. session_journal held 3,635,313 records, 3,635,016 of them terminal.output (9 GB). journal_subject_index had 21.8M rows, and journal_segments was empty. The daemon needed about 3 minutes of single-threaded CPU before it listened, and on that machine the daemon's start time stayed proportional to all the terminal output ever printed.

Measured causes, all open-time work that read the whole journal:

PRAGMA quick_check read every page of the file. A read-only run on that machine took more than 3 minutes by itself.
create_session_journal_schema re-ran INSERT OR IGNORE INTO journal_event_index ... SELECT ... FROM session_journal on every open, twice.
restore_agent_roster replayed every record since the roster cursor, and it decoded and SHA-256 hashed each output chunk. The cursor moves only on agent events, so a restart replayed all the output since the last agent event. fold_agent_roster did the same under the registry lock on the next agent event.

Fix

Event index: Appends write their journal_event_index row in the same transaction. So a full backfill runs only when the index table is new, and an existing index repairs only the rows past its highest sequence (a seek).
Integrity check: It runs PRAGMA quick_check('<table>') on each registry state table and its indexes. It skips the journal history tables (session_journal, journal_subject_index, journal_event_index, journal_segments, journal_checkpoints, journal_content_blobs), because they are append-only or sealed archives whose records are validated when they are read. The failure behavior is the same. The bundled SQLite (libsqlite3-sys 0.37) supports the table argument.
Roster replay: The roster ignores terminal output: its producer is the terminal runtime and its format is fixed (cmux.terminal-output.v1), so it is neither an agent-hook nor an agent-plugin event, and AgentRoster::apply returns before any mutation. A new session_journal_without_output_after(sequence, through, limit) reads only the other records, through two ranges of the (kind, sequence) index around terminal.output. Replay then sets the cursor to the target sequence. Sealed segments are decoded and filtered. Reducer semantics do not change.

Verification

The regression test registry_reopen_cost_does_not_scale_with_terminal_output_history counts page-cache misses on reopen (SQLITE_DBSTATUS_CACHE_MISS) after 16 and after 16Ki output records.

RED on the test-only commit: reopen read 27740 pages after 16Ki output records but 133 after 16, https://github.com/manaflow-ai/cmux/actions/runs/36375392810
GREEN on the fix: https://github.com/manaflow-ai/cmux/actions/runs/36376891975 (Linux and macOS tests plus clippy)

Existing suites on the fix commit, hosted Linux and macOS:

agent_roster: pass, https://github.com/manaflow-ai/cmux/actions/runs/36379384717
workspace_registry::tests: 122 pass, 1 fail (startup_repairs_legacy_terminal_close_dangling_resource_rows), https://github.com/manaflow-ai/cmux/actions/runs/36378307621
journal: 133 to 136 pass, 1 fail (mux::tests::startup_repairs_a_plugin_projection_lost_after_journal_commit), https://github.com/manaflow-ai/cmux/actions/runs/36380564266

Both failures are pre-existing. They fail in the same way on main plus only the new test, without this fix: https://github.com/manaflow-ai/cmux/actions/runs/36379510133 and https://github.com/manaflow-ai/cmux/actions/runs/36381468435.

Follow-up

This PR bounds what startup reads. It does not stop the file from growing. The root-cause fix is retention for terminal.output: seal segments automatically after each checkpoint (today seal_journal_segments runs only through the session.journal.segment.seal RPC, which nothing calls), prune journal_subject_index and journal_event_index for the sealed range (their append-only triggers block this today), and reclaim space with incremental vacuum.

Changelog

Fixed: cmux-tui daemons with a long terminal history now start in seconds instead of minutes.

🤖 Generated with Claude Code

^{Need help on this PR? Tag @codesmith-bot with what you need. Autofix is disabled.}

Summary by cubic

Bounds cmux-tui daemon startup by recent journal work instead of total journal size. A production registry with 3.6M terminal.output records (12 GB) took about 3 minutes of single-threaded CPU before the daemon listened.

Startup reads no longer span the whole journal

Appends write their journal_event_index row in the same transaction; a full backfill runs only when the index is new, and an existing index repairs only the rows past its highest sequence.
Open-time quick_check runs per registry state table and its indexes, skipping the append-only journal history tables whose records are validated when read.
Agent-roster replay skips terminal.output records (the roster ignores them), follows the (kind, sequence) index through the other kinds, and stands on the head; sealed segments are decoded and filtered. Reducer semantics are unchanged.
Adds a regression test that keeps reopen page-cache misses flat up to 16Ki output records.

This bounds what startup reads but does not stop the file from growing; retention and vacuum are separate follow-ups.

Summary by CodeRabbit

Performance
- Improved workspace startup and journal replay efficiency, especially for workspaces with large terminal-output histories.
- Reduced unnecessary journal-index work when reopening an existing workspace.
Reliability
- Workspace registry initialization now checks individual tables for integrity and reports an error if a check fails.