perf(kernel): faster column-to-byte conversion on long non-ASCII lines
Indexing JetBrains Exposed, whose docs folder carries minified scripts, takes a median 25.7 s instead of 37.0 s (31% faster, 5 alternating runs per build).
| Metric | Before | After | Change |
|---|---|---|---|
Exposed, codegraph init | 37.0 s | 25.7 s | 31% lower |
| nocodb (no long lines, control) | 6.74 s | 6.53 s | no clear change |
Median of 5 runs per build on d59ca2b3, baseline run twice in the same alternation (Exposed baseline against baseline: 0.3%; nocodb: 2.8%, so the reporting band is 5%). Every Exposed run on the new build (24.98 to 25.91 s) is below every baseline run (36.03 to 38.30 s). Node 24.21.0, same dist/, only the kernel binary differs.
What changes
The resolver converts a reference's column (UTF-16 units) into a byte offset many times per reference. #477 made that O(1) when the text before the column is ASCII. A line that has any non-ASCII character before the column still walked every character from the start of the line on each lookup. On a minified one-line script that is tens of thousands of characters per lookup. A perf profile of an Exposed index put 31% of all samples in that walk.
js_unit_to_byte now counts UTF-16 units over 64-byte blocks and walks characters only inside the block that holds the column. The block counts are u8 sums: the same count in usize does not vectorize and ran no faster than the old walk (a first version of this change measured +0.6% on Exposed for that reason; 600 KB line, microbenchmark: char walk 218 µs, usize sum 234 µs, u8 sum 13 µs).
The output is unchanged: Exposed 28,689 nodes, 82,842 edges and 36,202 call edges, nocodb 24,570 nodes, 48,763 edges and 15,039 call edges, with no call edge only in either build (scripts/index-metrics.mjs).
Checks
- Unit tests compare the new function with the character walk at every offset, including 4-byte characters and multi-byte characters on and around block edges (
cargo test --release, 70 passed, 1 ignored). - Full vitest suite on
cc22d774: 702 files, 8,950 passed, 39 skipped, 0 failed; kernel golden dumps unchanged. - README: checked the "About this fork" tables and the performance paragraphs; none quotes an Exposed index time or this path, so no row changes. CHANGELOG entry added under Fixes.
Not covered: lines made only of ASCII still pay one vectorized ASCII check per lookup (3.8 µs on a 600 KB line); a per-line cache would remove that too but needs an invalidation story and was not worth it at this size.
Summary by cubic
Faster column-to-byte conversion in js_unit_to_byte, which previously walked every character on a line when any character before the offset was non-ASCII — a per-lookup cost on minified one-line scripts. Indexing JetBrains Exposed now takes a median 25.7 s instead of 37.0 s (31% faster, 5 alternating runs per build) with identical output; the nocodb control with no long lines is unchanged.
Summary by CodeRabbit
- Performance
- Improved indexing performance for projects with long minified lines containing non-ASCII characters. Median indexing time decreased in testing for JetBrains Exposed; results for nocodb showed no change beyond run-to-run variation.