perf(index): a fresh mid-size index stops syncing the database inside its resolution commits
On a fresh index of a mid-size project, where the resolver pool starts part-way through, persisting resolved batches now takes 21–39% less time: ktor 3.66 s to 2.25 s, firefly-iii 1.60 s to 1.14 s (full index, Node 24, median of 3 alternating runs per build, identical graphs). During resolution the main thread syncs the disk 6 times instead of 145 on firefly-iii and 7 times instead of 204 on Exposed. Whole-index time, CPU and peak memory show no clear change on any of the six projects measured.
The chart's "no clear change" line is 10.3% of the baseline: the largest spread between before and after runs on any persist row. Each row is also judged in the tables below against its own project's spread.
What was wrong
A fresh index (fast init) starts in memory-journal mode. Below 150,000 pending references, the adaptive pool from #506 can start part-way through resolution. When it does, ensureWalForPool (src/codegraph.ts) switches the database to write-ahead logging so the pool's readers can share it. Nothing turned SQLite's default auto-checkpoint off at that point, and no write-ahead-log valve was armed. So from then on, every 1,000 pages of log made a batch COMMIT copy pages into the database file and sync it while the main thread waited. At or above the 150,000 gate this does not happen: the index switches to write-ahead logging before resolution, with auto-checkpoint at 0 and the valve running.
The change
In indexAll (src/codegraph.ts), a fast-init index now always sets auto-checkpoint to 0 and starts the WalCheckpointValve for the resolution phase. Before, it did this only above the 150,000 gate. While the database is still in memory-journal mode both have no effect. If the pool later switches to write-ahead logging, it inherits them. After that:
- The valve folds the log off the main thread past 256 MB and holds the writer at 512 MB, as it already does above the gate.
- The existing
finallyblock restores the previous auto-checkpoint value and journal mode. Maintenance runs the final checkpoint, also as before. foldWalForSnapshotis unchanged. The pool's reader snapshot still folds the log before it opens.
The source change is 16 lines added and 8 removed in src/codegraph.ts, including a comment in ensureWalForPool, plus one 37-line test.
Trade-off: the write-ahead log now grows until the valve folds it. Peak log size per run was 492 MiB on ktor (before: 10 MiB), 328 MiB on Exposed, 176 MiB on firefly-iii, 160 MiB on paperless-ngx and 70 MiB on bloc. pretix was 178 MiB both before and after. The valve caps the file at 1 GiB.
Crash safety: a fast-init index already counts as disposable until it completes. Its index_state stays indexing, and an interrupted run re-indexes from scratch. With write-ahead logging and synchronous = NORMAL, a process crash loses no committed batch. A power loss can drop the most recent batches but leaves a consistent database. A large log left behind by a crash is folded at the next open, which already happens above 64 MB.
Results
Before = fork/consolidated d161ef74; after = this change. The after build was measured twice per round: the change itself, plus an identical copy of it as a control arm. A change counts only when the medians differ by more than the floor and the ranges do not overlap; otherwise it is "no clear change". All runs used Node 24.22.0, 3 rounds with alternating builds, and were admitted only on a quiet host.
Persist time and main-thread commits come from instrumented runs (SQLite statement timing, 3 per build). For these, the floor is the larger of the before and after ranges.
| Project | Persist resolved batches, before (range) | After (range) | Benefit | Floor | Main-thread COMMIT time / writes, before → after |
|---|---|---|---|---|---|
| ktor | 3,664 ms (3,633–3,683) | 2,247 ms (2,232–2,270) | 39% less | 50 ms | 2,188 → 657 ms; 1,068 → 559 MiB |
| Exposed | 2,383 ms (2,379–2,482) | 1,539 ms (1,515–1,548) | 35% less | 103 ms | 1,344 → 427 ms; 676 → 364 MiB |
| firefly-iii | 1,601 ms (1,593–1,620) | 1,138 ms (1,114–1,160) | 29% less | 46 ms | 772 → 268 ms; 427 → 236 MiB |
| paperless-ngx | 1,435 ms (1,413–1,453) | 1,099 ms (1,055–1,145) | 23% less | 90 ms | 673 → 240 ms; 382 → 207 MiB |
| bloc | 640 ms (634–700) | 507 ms (477–511) | 21% less | 66 ms | 225 → 80 ms; 126 → 71 MiB |
| pretix (control: starts the pool before resolution) | 1,272 ms (1,254–1,324) | 1,287 ms (1,252–1,310) | no clear change (+1.2%) | 70 ms | 184 → 184 ms; 151 → 149 MiB |
The plain runs give whole-index time, CPU and peak memory (3 per build). The floor is the spread across all six runs of the change and its control copy.
| Project | Wall, before → after | Floor | CPU, before → after | Peak memory, before → after | Result |
|---|---|---|---|---|---|
| ktor | 13.81 → 13.89 s | 0.73 s | 52.67 → 51.90 s | 2,500 → 2,442 MiB | no clear change on all three |
| Exposed | 9.45 → 9.51 s | 0.87 s | 35.36 → 35.34 s | 2,490 → 2,479 MiB | no clear change on all three |
| firefly-iii | 5.19 → 5.07 s | 0.20 s | 15.67 → 15.72 s | 2,098 → 2,105 MiB | no clear change on all three |
| paperless-ngx | 3.86 → 3.75 s | 0.32 s | 12.30 → 12.24 s | 1,624 → 1,603 MiB | no clear change on all three |
| bloc | 2.29 → 2.29 s | 0.03 s | 6.91 → 6.94 s | 1,070 → 1,063 MiB | no clear change on all three |
| pretix | 9.53 → 9.46 s | 0.19 s | 32.65 → 32.58 s | 3,403 → 3,349 MiB | no clear change on all three |
Wall time does not move because the saving is split up and absorbed elsewhere. The resolution phase gets shorter: 270 ms on firefly-iii, 264 ms on paperless-ngx and 86 ms on bloc. The ktor and Exposed reductions (327 and 185 ms) are inside their floors. But maintenance gets 42–133 ms longer, because the final fold of a larger log now happens there. Resolution and maintenance together come out ahead by 160 ms on firefly-iii (floor 92 ms) and 43 ms on bloc (floor 30 ms); on the other projects the combined change is inside the floor. On ktor and Exposed the pool's own resolve time, not the commits, appears to set the pace.
A syscall trace of one run per project, compared with the same trace of the before build, shows:
- Main-thread syncs during resolution: firefly-iii 145 → 6; Exposed 204 → 7.
- Main-thread writes to the database file during resolution: firefly-iii 190 → 0 MiB; Exposed 310 → 0 MiB. The fold now writes on its own thread: 22 MiB on firefly-iii, 24 MiB on Exposed.
- Database-file writes across the whole run: firefly-iii about 607 → 481 MiB; Exposed about 664 → 391 MiB. A page changed many times is now copied into the database file once.
Output identity: on every accepted run, both copies of the change produce the same graph hash as the before build on the same project: ktor 57e9c5136b5e, Exposed b332758c901b, firefly-iii 496d20b7d5b6, paperless-ngx 2cca11fd5dbc, bloc c49b5b74c4a9, pretix bc3469fcdcfc. Plain runs: 72 accepted, 1 rejected and re-run (memory pressure from other processes). Instrumented runs: 36 accepted, none rejected.
Checks
All ran on the head of this branch after rebasing onto 87e304a5:
__tests__/wal-deferral.test.ts, new case "fast init: a resolver pool that boots mid-resolution". It starts the pool's write-ahead-log switch inside resolution and records the database state there. It fails on the base (auto-checkpoint 1000, no valve) and passes with the change (auto-checkpoint 0, valve armed; afterwards the journal mode iswaland auto-checkpoint is back at 1000).npx tsc --noEmit: exit 0.npm run build:kernel, includingcargo clippy --release --lib -- -D warnings: exit 0.npx vitest run __tests__/kernel-golden-dumps.test.ts: 8 passed; no.dumpfile changed.npm run check:agent-docs: exit 0.- Full suite
npx vitest run: 746 files, 9,263 passed, 40 skipped, exit 0.
The resolution precision gate was not run. Resolution logic is unchanged, and the graph hashes are identical on all six projects.
The benchmark compared builds of d161ef74. Since then the base has gained #517, a kernel memory change for Kotlin, and #516, a review-bot setting. This change does not touch either, but the numbers above were not re-measured on the rebased head.
README and docs
- README "About this fork" rows checked:
- The adaptive-pool paragraph: 11–48% faster at 56–132% more memory.
- The 2026-10-15 mid-size paragraph: ktor 13.90 s and Exposed 9.81 s.
- The fork-vs-upstream chart rows for pretix.
None of them moves, since whole-index time, CPU and memory show no clear change. The README is unchanged.
- CHANGELOG: one entry under Unreleased → Fixes.
docs/benchmarks/adaptive-pool-engage-2026-10-15.md: its Limits section said a mid-run pool start has no checkpoint valve. A sentence now points to this change and its log size.
Not covered
- pretix's remaining resolution-time gap to upstream. pretix starts its pool before resolution, so this change does not touch it.
- A one-file sync, and an index over an existing write-ahead-log database. Both already armed the valve before this change.
- Hosts other than one Linux machine; Bun.
Summary by cubic
On a fresh mid-size index where the resolver pool boots part-way through, the switch to write-ahead logging brought SQLite's default 1000-page auto-checkpoint into the batch commits, so pages were copied and synced on the main thread during resolution. A fresh index now keeps auto-checkpointing off for the whole resolution phase and arms the write-ahead-log valve up front; both settings are inert until a WAL exists, so sequential runs are unchanged.
Persisting resolved batches is 21–39% faster on the five projects that take this path (ktor 3.66 s to 2.25 s, firefly-iii 1.60 s to 1.14 s), and main-thread syncs during resolution drop from 145 to 6 on firefly-iii and 204 to 7 on Exposed. Whole-index time, CPU and peak memory show no clear change, and graphs are identical.
Trade-offs
- The write-ahead log now grows until the valve folds it: up to 492 MiB on ktor (previously near 10 MiB), capped at 1 GiB.
- A fresh index remains disposable until it completes; an interrupted run re-indexes from scratch, and crash safety is unchanged.