perf(resolution): mid-size projects index faster by starting the resolver pool part-way through
Full index of five mid-size projects finishes 11–48% faster, for example JetBrains Exposed (110,854 pending references) 18.5 s to 9.6 s, at 56–132% more peak memory; the graph is identical in every run.
Below 150,000 pending references the resolver used to stay on one thread for the whole run (the gap #490 documented). Now, after each main-thread batch, the loop projects the time the main thread would still need (measured kernel resolve ms per reference × references left) and boots the existing pool when at least 20,000 references remain and the projection is at least 400 ms. Because the pool's workers read the live database, a rollback-journal (fast-init) database is switched to WAL first, and the start is declined if that fails. An explicit CODEGRAPH_PARALLEL_RESOLVE_MIN turns the projection off. The pool's own preflight is unchanged, so it still declines on hosts with fewer than three cores or little memory. The 20,000 / 400 ms constants are upstream's, untuned here.
| Corpus (pending refs) | Wall, before → after | Benefit | Peak memory | CPU |
|---|---|---|---|---|
| Alamofire (22,684) | 1.65 s → 1.41 s | 15% lower | 660 → 1,056 MiB (+60%) | 3.4 → 4.8 s |
| bloc (37,140) | 2.57 s → 2.30 s | 11% lower | 682 → 1,064 MiB (+56%) | 5.2 → 6.9 s |
| koel (84,654) | 4.93 s → 4.11 s | 17% lower | 789 → 1,498 MiB (+90%) | 9.6 → 12.4 s |
| paperless-ngx (102,380) | 5.37 s → 3.95 s | 26% lower | 848 → 1,596 MiB (+88%) | 9.1 → 12.6 s |
| JetBrains Exposed (110,854) | 18.50 s → 9.55 s | 48% lower | 1,077 → 2,497 MiB (+132%) | 23.2 → 35.8 s |
| gin (13,892, control, pool never starts) | 0.97 s → 0.97 s | no clear change | 404 → 401 MiB | 1.5 → 1.5 s |
Median of 3 interleaved runs per arm (order alternating), Node 24.22.0, fresh index each run, host lock held. In the five engaged rows the after range does not overlap the before range. There is no separate A/A arm; the three-run spread of the before arm is the noise reference. Identity gate: md5 over every nodes row (all columns except updated_at), every edges row (all columns except the autoincrement id) and the unresolved_refs count per status is equal between arms for every corpus and round. Method, ranges and limits: docs/benchmarks/adaptive-pool-engage-2026-10-15.md.
Verification on 960e5058: tsc --noEmit clean; vitest resolver-pool-engage (new, four cases: the engage decision and the explicit-threshold override), function-ref and resolution, resolver-pool-sizing, readme-sync pass; npm run check:agent-docs passes. Kernel golden dumps are unaffected (no extraction or resolution change; edges identical).
Limits: the memory cost is the same trade the 150,000-reference default already makes. Alamofire projects about 500 ms against the 400 ms bar, so it is close to the edge. pretix and projects between 111,000 and 150,000 references were not re-measured; Linux, one host.
README rows checked: the two pool-threshold statements in "About this fork" (the paragraph under the 2026-10-14 fork-vs-upstream results, and the "Full index" bullet under hardware tiers) are updated; the 74-comparison chart and its numbers are unchanged because those corpora were measured before this change. CHANGELOG [Unreleased] and the metrics ledger (the "adaptive mid-loop engage is gone" line) are updated.
Summary by cubic
Full index of mid-size projects finishes 11–48% faster (JetBrains Exposed 18.5 s → 9.6 s) by starting the resolver pool part-way through the run, at 56–132% more peak memory; the graph is identical in every run.
Below 150,000 pending references the resolver used to stay on one thread. After each main-thread batch, the loop now projects the remaining kernel resolve time from the measured per-reference rate and boots the pool when at least 20,000 references remain and the projection is at least 400 ms. The pool's workers read the live db, so a rollback-journal (fast-init) database is switched to WAL before the pool boots and the start declines if that fails. An explicit CODEGRAPH_PARALLEL_RESOLVE_MIN replaces both thresholds and turns the projection off. Projects that never reach the threshold are unaffected.
Benchmarks
- Alamofire 1.65 → 1.41 s, bloc 2.57 → 2.30 s, koel 4.93 → 4.11 s, paperless-ngx 5.37 → 3.95 s, JetBrains Exposed 18.50 → 9.55 s; medians of 3 alternating runs, candidate ranges not overlapping baseline.
- gin (13,892 references, control) is unchanged; node and edge rows are md-identical across all corpora and runs.
- A fresh index below the gate that starts the pool mid-run has no checkpoint valve; one instrumented Exposed run peaked at a 9.5 MB write-ahead log against a 127.7 MB database, pool started once.
- The memory cost is the same trade the 150,000-reference default already makes; the 20,000/400 ms constants are upstream's and untuned.
Summary by CodeRabbit
- Performance
- Resolver processing can switch from single-threaded work to multiple threads partway through a run when at least 20,000 references remain and the estimated remaining single-threaded work is at least 400 ms. This applies below the existing 150,000-reference threshold.
- Benchmarks show indexing time improvements of 11–48% on projects where the pool engaged, alongside 56–132% higher peak memory use. Projects below the reference threshold are unchanged.
- Setting
CODEGRAPH_PARALLEL_RESOLVE_MINdisables adaptive pool startup.