PR #28771 - manaflow-ai/cmux

Every running Codex writes state_<n>.sqlite about once a second. Each of those writes made acpmux's chat watcher (acpmux-chat-watch) read the whole threads table again. A row holds its first prompt three times (title, preview, first_user_message, about 29 KB each on a busy machine). On a lane VM with 3,285 threads, one read was 392 MB and about 100 ms, even though only about two rows had changed. That thread read 40-60 MB/s around the clock (/proc/<pid>/task/<tid>/io: 5.1 GB in 90 s, in 4 KB SQLite page reads).

Leo's words: "how much RAM, how many CPU cores are we taking with all these things? Are there ways to make things more efficient?" and, on 2026-10-31, "do we have more productionized stuff wrt the daemon".

Part of manaflow-ai/cmuxterm-hq#4870 (daemon memory and idle CPU).

What changes

cmux-chat-index's Codex adapter keeps the last read of each Codex home's database (adapters/codex/threads.rs, at most 16 homes). Each read works like this:

Keys first. It reads every row's key in one snapshot:
- its created and updated times;
- the full value of every small column the query reads or filters on (name, cwd, originator, rollout_path, archived, has_user_event, thread_source, agent_nickname, agent_role);
- only the byte length of the three prompt columns. SQLite answers those from the row header, and Codex's auto_vacuum pointer map lets it pass the prompts' overflow pages. On the real DB the keys cost 25 MB.
Changed rows only. It reads again only the rows whose key changed, or whose spawn edge came or went. A thread counts as spawned when SQLite's own IN matches it (SELECT id FROM threads WHERE id IN (SELECT child ...)), the same affinity rule the filter's NOT IN uses. These reads use the same SQL filters as before (... WHERE id IN (...) AND <filters>). A keyed row the filtered read does not return is not a chat.
Whole reads. The whole table is read again:
- on a cache miss, a replaced file (dev/inode, or the creation time off Unix), or a change to the two tables' schema (their sqlite_master text, so a column, type or collation change counts);
- at least every 5 minutes. A same-length rewrite of a prompt column with no time bump is the one change a key misses, and it shows within that time.

Review round 1 (Astra, BLOCK at 2b0b02a) listed seven P2s. All are addressed in 547d5a3:

same-length name renames;
the created time missing from the key;
Rust-side filters with different SQLite affinity semantics (the filters are SQL again);
excluded children being decoded;
an unbounded cache;
file identity on Windows;
a test oracle sharing the cache. The fresh read now calls the doc-hidden forget_codex_reads first.

Round 2 (BLOCK at 547d5a3) listed two P2s, fixed in b6dc05f (red test e32c105):

spawn-edge children kept only text ids, so an integer child under numeric affinity diverged from NOT IN;
only the projection SQL was compared, so a column whose filter changes with no row changing (for example has_user_event added back) stayed stale until the 5-minute read.

Round 3 (BLOCK at b6dc05f) listed two P2s on the schema check, fixed in 2216320 (red test 8db59a6):

the columns were read before the snapshot began, so a column added in between could be recorded with the new schema text but no filter. The transaction now starts first;
table names were matched by exact case, while SQLite resolves threads against Threads. They are matched with lower().

Performance

Same harness for both: a copy of the lane VM's Codex state DB (3,285 threads, 2,135 chats, auto_vacuum = 2 as Codex makes it) and a throwaway example binary built at the base and at the head. Each burst bumps one thread's updated_at_ms and calls ChatIndex::paths_changed(wal), 21 bursts per run, two runs each. Bytes are /proc/thread-self/io rchar.

metric	before	after	delta	harness or run link
bytes read per burst (median)	392.49 MB	25.32 MB	-93.5%	DB copy, 21 bursts x 2 runs
burst time (median)	92-131 ms	17-25 ms	about -80%	same
first whole read, bytes	392.5 MB	417.7 MB	+6% (the keys)	same
chat list	2,135 chats	identical	none	serde digest: each head run's first read equals the previous base run's result after its bursts, and vice versa

On the live daemon the watcher burst comes about every 2-7 s while any Codex runs, so this removes most of the steady 40-60 MB/s of page-cache reads the watcher thread made.

No new timer, observer or loop. The cache holds one entry per thread per Codex home (16 homes at most). Each read takes it out and puts it back only when the read succeeds, so a failed read leaves none and the next read is a whole read.

Tests

Red on the base (f7d2ba37a0d): cargo test -p cmux-chat-index --test codex_store_reads. a_thread_update_reads_that_thread_not_the_table fails with "one changed thread read 26281709 bytes of a 37966240-byte database". The bound is now 1/4 of the database; the head reads ~11% (4.2 MB), base 69%.

Red at 547d5a3 (e32c105): a_schema_or_affinity_change_matches_a_fresh_read fails "after the column is added", and with the schema fix only, "after an integer spawn edge".

Red at b6dc05f (8db59a6): the same test fails "after the column is added" on a table renamed Threads.

Green on the head (2216320):

cargo test -p cmux-chat-index: all pass.
- every_store_change_matches_a_fresh_read covers, each compared with a fresh whole read: title, name, a same-length name, the created time only, cwd, archive (all with no time bump), subagent, spawn edge added then removed, delete, insert.
- a_schema_or_affinity_change_matches_a_fresh_read: a table renamed Threads, a dropped and re-added has_user_event, and an integer spawn edge in an integer child column.
- a_same_length_rewrite_shows_after_the_full_read_bound
cargo clippy -p cmux-chat-index --all-targets -- -D warnings: clean.
On cmux-ci, acpmux tests/chats.rs passed 10/10 at 2b0b02a (run 41160256159) and at 547d5a3 (run 41161762386), including an_idle_codex_state_db_is_not_read_again_and_again. The red mark on those runs is the deliberate exit 1 that prints the log.

Observed / Not run

Observed: the above, plus the real-DB before/after with equal chat lists.
Not run: a live daemon on this build. The Performance table was measured at 547d5a3. b6dc05f adds one children query per burst that walks thread_spawn_edges and probes the threads primary-key index, not its rows; it was not measured again.

Risk: normal. A read-only change to how a harness's SQLite store is read; nothing is written to the harness's folder. No UI change, so no capture.

Changelog

Fixed: acpmux no longer rereads Codex's whole state database each time a Codex chat writes, which cut a steady 40-60 MB/s of reads.

Agent session: agent session

🤖 Generated with Claude Code