PR #519 - bompus/codegraph

Indexing pretix is 4.9% faster, a median 10.17 s down to 9.67 s with its parse phase 16% shorter, and indexing supabase uses 6.7% less CPU (full index on Node 24, 3 alternating runs per build, identical graphs).

The chart shows pretix. Its "no clear change" line is 3.2% of the baseline, the largest spread between two identical builds of this change on any pretix row. supabase is in the table below, where each row is judged against its own corpus's spread.

What changed

The fork finds file dependencies from CommonJS calls such as require('./x') by walking a tree-sitter tree, so that a shadowed require (a parameter, a local, an import named require) and string or comment text are left out. The kernel's extraction tree is not available on the JavaScript side, so commonJsRequireRefs parsed every JavaScript or TypeScript file that contains the word require a second time. Few of those files have such a call: 4 of 112 in pretix, 27 of 1,126 in supabase, 17 of 4,499 in n8n. The rest say typeof require, required:, __webpack_require__ or require('lodash').

Now (src/extraction/commonjs-requires.ts):

mayRequireModulePath reads the text at each require: an optional ?., (, any empty argument slots, then a quoted literal whose text is a relative path, a # import or a path with a slash, which is what the walk accepts. It skips whitespace and // and /* */ comments the way the grammar does. Anything it cannot read counts as a possible call and the walk runs as before: non-ASCII or control characters between tokens, , an unclosed comment, TypeScript type arguments, an escape, ${ in a template, a line break in a quoted string.
That reading only holds for a file that parses without errors, because tree-sitter's error recovery can build a call from require@('./x'). The kernel already computes has_error() on its tree; codegraph-kernel/src/buffers.rs now writes it into byte 1 of the result metadata, which was zero padding. An older kernel leaves the byte 0, read as "unknown", so the buffer layout stays at version 3. src/extraction/kernel/index.ts reads it, and also requires that the kernel parsed the same text (a Flow file is blanked before parsing).
commonJsRequireRefs skips the second parse only when the kernel reports a clean parse and the text has no possible call. The kernel path and the parse worker's raw path pass the flag; the remaining caller passes nothing and keeps the old behavior.

Results

Full index (codegraph init), before = fork/consolidated d161ef74, after = this change on that base (the rebase onto #516, #517 and #518 adds commits that touch none of these files). Medians of 3 runs, builds alternating within each round, with a third build that is an identical copy of this change. A change counts when the medians differ by more than that corpus's floor (largest minus smallest of the six runs of the two identical builds) and the ranges do not overlap; otherwise "no clear change". All runs: Node 24.22.0, systemd-run --user --scope -p MemoryMax=16G, admitted only on a quiet host, rejected when other processes used more than 2 cores.

Corpus	Measure	Before (range)	After (range)	Benefit	Floor
pretix (Python + JS, 1,467 files)	Wall time	10.17 s (10.02-10.43)	9.67 s (9.49-9.73)	4.9% faster	0.24 s
pretix	CPU time	34.12 s (33.83-35.43)	32.76 s (32.60-33.15)	4.0% less	0.59 s
pretix	Parse phase	2.48 s (2.42-2.48)	2.07 s (2.04-2.08)	16.4% shorter	0.08 s
pretix	Peak memory	3,384 MiB	3,350 MiB	no clear change (-1.0%)	146 MiB
supabase (React + Next.js + TS)	Wall time	21.34 s (19.77-21.48)	20.39 s (19.06-20.83)	no clear change (-4.5%)	1.90 s
supabase	CPU time	74.95 s (72.25-75.04)	69.92 s (68.23-71.55)	6.7% less	4.43 s
supabase	Parse phase	6.19 s (6.12-6.33)	5.70 s (5.50-5.85)	7.9% shorter	0.42 s
supabase	Peak memory	3,832 MiB	3,829 MiB	no clear change (-0.1%)	136 MiB

"Parse phase" is the indexer's own parse-loop timing (CODEGRAPH_SYNTH_TIMINGS). supabase's wall time moved by about half its floor: its third round ran on a quieter host for every build (all four builds were 1-2 s faster than in rounds one and two), which widened the spread.

Where the time went: CPU profiles of one extra run per build and corpus (V8 --cpu-prof on every thread; these runs are not among the timing samples) put the require scan (commonJsRequireRefs, summed over threads) at 1.14 s before and 0.28 s after on pretix, and 2.12 s before and 0.11 s after on supabase. Of the 0.28 s left on pretix, 0.20 s is the second parse of files that still need it.

Graphs are identical: on every accepted run, the after build and its identical copy produce the same graph hash as the before build on the same corpus (pretix bc3469fcdcfc, supabase f41125aee6df). 24 runs accepted, 1 rejected and re-run (an upstream pretix run where other processes used 2.4 cores).

Against upstream (e8ec7fd8, same runs): pretix's index time goes from 21% above upstream's (8.39 s) to 15% above it. supabase's index time stays within the spread of upstream's (19.98 s), and its CPU time goes from 3.7% below upstream's (77.82 s) to 10.2% below it.

The text check was also compared file by file against the old walk through the real kernel flag, on every JavaScript or TypeScript file containing require in ten projects (pretix, n8n, supabase, nocodb, trezor-suite, fastify-demo, nest, discourse, koel, halo): no file's references changed. A generated-source fuzz of 1,000,000 cases (3,579 of them parse cleanly and yield a reference) changed no output.

Checks

On the final head f6509ce6 (rebased onto 0e6f7e30):

npm run build:kernel (clippy included) and npm run build: exit 0.
UPDATE_GOLDEN=1 npx vitest run __tests__/kernel-golden-dumps.test.ts: exit 0, 8 passed, .dump diff empty.
npx tsc --noEmit, npm run check:agent-docs: exit 0.
npx vitest run: exit 0, 747 files, 9,268 passed, 40 skipped (includes readme-sync, 6 passed, and the new test).

On the code of 3d0d8b42 (the same change before the rebase):

cargo test --release --lib buffers: exit 0, 2 passed.
New __tests__/commonjs-require-prefilter.test.ts: 5 passed on Node and on Bun 1.4.3; on the base d161ef74, 3 of the 5 fail. It checks that a clean file that only mentions require is not parsed again; that require /* c */ ?.(...), import x = require(...), require<T>(...) and require(, './d') still are; that a file with a syntax error keeps the walk; a property test over 6,000 generated sources (output equals the walk's on every clean one); and the kernel flag itself. With the empty-slot skip removed, the property test fails.
Existing CommonJS tests: pass.
Precision gate not run: no resolution change (golden dumps and graph hashes identical).

README rows checked

"About this fork" chart and summary paragraph, and the Full index table rows for pretix and supabase: a dated fork-vs-upstream run with its own builds. This change would move pretix's row (the fork's gap against upstream narrows from about 21% to 15% in these runs) and supabase's CPU, not its wall time. Not edited: updating them needs the full fork-vs-upstream benchmark re-run, left for the next scheduled one.
Feature tables: no user-visible feature, command or setting changes.
CHANGELOG: one entry under Unreleased → Fixes.

Not covered

Moving the walk into the Rust kernel, which would drop the second parse for the files that still need it: the references would then live in the kernel's tables and the golden dumps, a larger change.
The generic extractor (src/extraction/tree-sitter.ts), used when the native walker does not handle a file or is switched off with CODEGRAPH_KERNEL=0, still parses again.
One-file sync and Bun timings were not measured.

Summary by cubic

Speeds up indexing projects that mix in JavaScript or TypeScript by skipping the redundant second parse of files that mention "require" without a require('./x') call. A full index of pretix drops from a median 10.17 s to 9.67 s (parse phase 2.48 s → 2.07 s); supabase uses 6.7% less CPU. Graphs are identical.

What changed

The CommonJS require pass re-parsed every JS/TS file containing "require", though most only contain typeof require, required:, __webpack_require__, or prose. A text check now runs first, and the second parse happens only when the text could hold a require call with a module path argument, or when the file has a syntax error.
The check is only sound on a clean parse, since tree-sitter error recovery can turn require@('./x') into a call. The kernel now reports its parse status in a spare metadata byte; an older kernel leaves it 0, which reads as "unknown" and keeps the old behavior, so the buffer layout stays at version 3.
Files still parsed a second time are those with a possible call and anything handled by the generic extractor (CODEGRAPH_KERNEL=0).