Index store: one shared store replaces the trigram, SQLite call-graph and views indexes (0.64)
This is the 0.64 index-store cutover. Text search, the call graph and meaning search move to one index store per repository, shared by its worktrees and clones; the old trigram index, SQLite call-graph store and content-addressed views are deleted (about 215,000 lines net).
What changes for users
- Each repository builds its new indexes once on first use: text in about a second, the call graph in under a minute for large projects. Worktrees of one repository share them, and a new worktree starts from a sibling's indexes and processes only the files that differ.
- Meaning search reuses old vectors for unchanged files embedded with the same model.
- Old index folders stay on disk;
npx @cortexkit/aft cache prune-legacylists and removes them. views.enabled,views.callgraph_copiesandworktree.ram_overlayshow a removal notice;views.idle_checkout_hoursbecomesidle.checkout_hours. Config names retired in 0.62 or earlier are refused in user config and load with a notice in project config.trace_toandimpactlabel each root (main, test, attribute, public API, no caller found).
Measured on AFT's own repository (7,300 files)
Call-graph build 207 s to 17 s; index disk 1.7 GB to 0.3 GB; typical edit catch-up 4.7 s to 0.1 s; dead code first run 72 s / +4.25 GB to 6-8 s / +0.8 GB; an edited worktree's call graph median 3 MiB between publications; a new worktree at the same commit under a second instead of about 160 CPU-seconds.
Verification
- Full Rust suite on Linux: all pass except the 12 native-secret-shell tests and one rust-analyzer test the build server can't run (both fail the same way on main).
- Incremental call graph equals a cold build after each of 430 real edits (edit, reuse, extract and checkout gates).
- Dead-code findings unchanged across a 91-project corpus; search-quality benchmark rows byte-identical except five declared pattern-lane candidate counts (
land-index-store-0.64.json). - macOS and Windows compile with warnings as errors.
The release notes are in review; the version bump and tag follow through scripts/release.sh after this merges.
Summary by cubic
This is the 0.64 index-store cutover: text search, the call graph, meaning search, dead code and TODO counts all move to one index store per repository, shared by its worktrees and clones. The old trigram index, SQLite call-graph store and views are deleted (about 215,000 lines net). The data layer is always on: there is no development switch, and configure no longer builds or opens today's indexes at all.
What changes for users
- Each repository builds its new indexes once on first use; worktrees share them, and a new worktree starts from a sibling's indexes and processes only the files that differ.
- Meaning search reuses stored vectors for unchanged files embedded with the same model, and a repository is filled by one process at a time so no chunk is embedded twice.
- Editing a config file (Cargo.toml, tsconfig.json, package.json) rebinds only the files it changed: crate roots, found config files and workspace package names are tracked per name, not per workspace member walk.
- A repository whose index-store files a newer build wrote is refused with
storage_requires_newer_reader, naming the file and version, instead of being rebuilt over them. A store an older build published still serves text; its call graph counts as not built and the checkout's own bind builds it again. - Old index folders stay on disk;
npx @cortexkit/aft cache prune-legacylists and removes them. views.enabled,views.callgraph_copiesandworktree.ram_overlayshow a removal notice;views.idle_checkout_hoursbecomesidle.checkout_hours.- Config names retired in 0.62 or earlier are refused in user config; a project config carrying one still loads, translating the name with a notice.
trace_toandimpactfollow callers to real roots and label each root: main, test, attribute, public API, or no caller found.- Index-store background work (builds, meaning fills, refreshes) runs in process-wide pools, so a fill round waiting on the embedding backend no longer holds every other root's builds and tool calls.
- Minified bundles (
*.min.js, or JS/CSS with almost no line breaks) are still searched by grep, but are not parsed for the call graph, cut into meaning-search chunks, or counted as files the call graph fails to cover. - Dead code and unused exports run on the index store's own integer graph, with per-file facts extracted at index time; TODO counts come from stored markers, and both list their files from the store's checkout instead of walking the project.
- Automatic dead code and unused exports follow their own schedule: they wait for a published call graph, run on one root at a time, and at most once every 30 minutes while edits arrive.
aft_searchreads each candidate file once per search, no longer serializes concurrent searches' query embeddings, and matches file-name searches against the index's path list instead of walking the project.
Measured and verified
- On AFT's own repository (7,300 files): call-graph build 207 s to 17 s, index disk 1.7 GB to 0.3 GB, typical edit catch-up 4.7 s to 0.1 s, new worktree at the same commit under a second instead of about 160 CPU-seconds, and a busy worktree's call graph holds about a tenth of the memory after a broad edit (about 120 MB instead of 1 GB).
- The store keeps at most 1,024 index files open and one descriptor per segment file, so busy daemons stay below macOS's open-file limit; decoded call-graph files share a 256 MB bounded cache; a finished build returns its freed pages to the allocator, so resident memory settles near the working set; the memory census carries the meaning fill's own timings into daemon health.
- A refresh publishes once 256 files have been re-resolved since the last publication, so a broad edit's re-resolved files are released instead of being held until the checkout goes quiet.
- Incremental call graph equals a cold build after each of 430 real edits; dead-code findings unchanged across a 91-project corpus; search-quality benchmark rows byte-identical except five declared pattern-lane candidate counts.
- Replaying 6,956 recorded callers/call_tree/trace replies through the new store matches main's corpus except 34 reviewed differences (C++ type-matched method calls, macro-template calls and macro-generated self calls now resolve).
- Full Rust suite passes on Linux except the 12 native-secret-shell tests and one rust-analyzer test the build server can't run (both fail the same way on main); macOS and Windows compile with warnings as errors.