perf(tokenizer): scan with indexOf in fastForwardTo

AB
2mo ago
+634-4

Tokenizer.fastForwardTo scans for its sentinel one character at a time. This PR replaces the loop with a single String.indexOf, which V8 compiles to a vectorised memchr.

Measured impact

Tokenizer time over 258 real-world pages:

Markup profileTokenizer time
attribute-heavy−31.6%
script-heavy−43.6%
text-heavy−45.4%

indexOf has a fixed call overhead: about 12 ns, against 2.75 ns for the loop. Below a skip distance of 4–5 characters it therefore loses:

  • On real pages, about 28% of calls are that short, but they cover only 1% of the characters skipped.
  • The longest 0.5% of calls (script and style bodies, long attribute values) cover 46%.

I also modelled probing a few characters before calling indexOf. Every probe length came out slower than plain indexOf.

Known limit: markup where nearly every skip is 3 characters or fewer (e.g. <b>a</b><i>b</i> repeated) gets about 5% slower. The tipping point is around 80% short calls, which looks like generated files rather than real pages. The densest page in the corpus sits at 28% short calls and still gains.

Safety

  • Differential test harness (first commit): it keeps the current loop as a reference implementation and checks that the new method matches it on return value, resulting index and thrown exception. The cases cover:
    • start positions outside the buffer;
    • the empty buffer;
    • non-zero offsets;
    • astral-plane content.
  • Guard for c > 127: the needle table only covers 0–127. Without the guard, indexOf(undefined) would silently search for the literal string "undefined". Today every caller passes a markup character, but the type does not enforce it. The guard's measured cost is nil.

npm test (vitest + eslint + tsc + biome) passes on current master: 210 tests.

We have been running this change in production at Fasterize through a fork of 10.1.0.

🤖 Generated with Claude Code

https://claude.ai/code/session_017qfYgt9uWFLNVjYPeyZL7r

Summary by CodeRabbit

  • Performance
    • Markup parsing can now locate certain characters in input more efficiently, which may improve processing speed for applicable content.
  • Bug Fixes
    • Preserved correct handling of non-ASCII characters and edge cases while scanning input, including when the requested character is not found.