perf(tokenizer): scan with indexOf in fastForwardTo
Tokenizer.fastForwardTo scans for its sentinel one character at a time. This PR replaces the loop with a single String.indexOf, which V8 compiles to a vectorised memchr.
Measured impact
Tokenizer time over 258 real-world pages:
| Markup profile | Tokenizer time |
|---|---|
| attribute-heavy | −31.6% |
| script-heavy | −43.6% |
| text-heavy | −45.4% |
indexOf has a fixed call overhead: about 12 ns, against 2.75 ns for the loop. Below a skip distance of 4–5 characters it therefore loses:
- On real pages, about 28% of calls are that short, but they cover only 1% of the characters skipped.
- The longest 0.5% of calls (script and style bodies, long attribute values) cover 46%.
I also modelled probing a few characters before calling indexOf. Every probe length came out slower than plain indexOf.
Known limit: markup where nearly every skip is 3 characters or fewer (e.g. <b>a</b><i>b</i> repeated) gets about 5% slower. The tipping point is around 80% short calls, which looks like generated files rather than real pages. The densest page in the corpus sits at 28% short calls and still gains.
Safety
- Differential test harness (first commit): it keeps the current loop as a reference implementation and checks that the new method matches it on return value, resulting
indexand thrown exception. The cases cover:- start positions outside the buffer;
- the empty buffer;
- non-zero offsets;
- astral-plane content.
- Guard for
c > 127: the needle table only covers 0–127. Without the guard,indexOf(undefined)would silently search for the literal string"undefined". Today every caller passes a markup character, but the type does not enforce it. The guard's measured cost is nil.
npm test (vitest + eslint + tsc + biome) passes on current master: 210 tests.
We have been running this change in production at Fasterize through a fork of 10.1.0.
🤖 Generated with Claude Code
https://claude.ai/code/session_017qfYgt9uWFLNVjYPeyZL7r
Summary by CodeRabbit
- Performance
- Markup parsing can now locate certain characters in input more efficiently, which may improve processing speed for applicable content.
- Bug Fixes
- Preserved correct handling of non-ASCII characters and edge cases while scanning input, including when the requested character is not found.