perf(server): find in a long thread no longer re-parses every message on each keystroke
The server's find call is about 6x faster after the first keystroke on long threads: 600–750 ms per keystroke before, 100–110 ms after, on the three largest threads in a real database (2,000–2,700 searchable messages each).
Problem
Each find scan visits every message and looks its parsed text up in a cache that held 512 entries. A thread with more than 512 distinct messages is walked in order, so each lookup evicts an entry the same scan needs next, and nothing is ever found in the cache. Every keystroke (each is a new query) and every next/previous match press while the thread is streaming re-parses the whole thread.
Change
Raise the parsed-text cache capacity from 512 to 8192 (ThreadFind.ts), with a comment saying why it must exceed the largest thread. Add a regression test: on a 600-message thread, a second query that differs from the first re-parses nothing (600 parses on the second query before the change, 0 after).
Scope and approval
A one-constant fix for a performance defect, with a test that fails without it. No output changes: match counts and positions are identical between the two builds in every measured case. I think it qualifies for the small-focused-fix exception; happy to open a discussion first if you would rather.
Verification
Read-only snapshot of a real 15 GB database, three largest threads, ThreadFind called in process (so these are server times, not measured in the UI). Before = upstream/main, after = this branch. Five interleaved pairs of separate processes (order alternated), host load 3.4–5.3; queries are 3–6 character prefixes of words in each thread, every typing call a new query. Median per call:
| Thread (searchable messages) | Scenario | Before | After | Benefit |
|---|---|---|---|---|
| 2,028 | typing, after the first keystroke | 600 ms | 104 ms | 5.8x |
| 2,028 | next match while the thread streams | 596 ms | 105 ms | 5.7x |
| 2,726 | typing, after the first keystroke | 751 ms | 112 ms | 6.7x |
| 2,726 | next match while the thread streams | 764 ms | 107 ms | 7.1x |
| 2,359 | typing, after the first keystroke | 731 ms | 103 ms | 7.1x |
| 2,359 | next match while the thread streams | 732 ms | 99 ms | 7.4x |
| all three | next match on a quiet thread (cached result) | 0.2–0.3 ms | 0.2 ms | no clear change |
- The first keystroke of a find session is unchanged (about 830 ms before, 820–930 ms after): the cache starts empty. Entries expire one minute after they were parsed, whether or not they were read since, so a session longer than a minute pays one full parse again.
- Heap after typing grew by about 4–9 MB more than before on these threads (0.6–0.8 M characters of text each). The cache is shared across threads and bounded by entry count, so the worst case is larger: entries hold their text and parsed segments, and 8192 entries of large messages could reach hundreds of MB. Expired entries are only dropped when their key is read or newer entries push them out.
- Threads above 8192 messages still cycle the cache. The separate cache for texts over 32,768 characters keeps its capacity of 32.
- Check:
vp test run apps/server/src/orchestration-v2/ThreadFind.test.ts(24 tests pass); the new test fails on the old capacity with "expected 600 to equal 0". Not checked: UI-side effect of the shorter server time, other machines or data, Bun.
Claude Sonnet 5.5 in Claude Code, run through T3 Code.
Summary by cubic
Makes find in long threads about 6x faster after the first keystroke. The parsed-text cache was too small (512 entries) to hold a whole thread, so each scan evicted entries it still needed and re-parsed all 2,000–2,700 messages per keystroke; raising the capacity to 8192 keeps parses cached across queries and cuts median time from 600–750 ms to 100–110 ms. Adds a regression test asserting a second query on a 600-message thread re-parses nothing (600 parses before, 0 after).
Side effects
- Heap grows roughly 4–9 MB more per large thread while typing; the shared cache is bounded by entry count, so 8192 entries of large messages could reach hundreds of MB.
- Threads above 8192 messages still cycle the cache, and the separate cache for texts over 32,768 characters is unchanged.
- No output changes: match counts and positions are identical in all measured cases.