fix(host-service): bound transcript line reads so one corrupt file cannot kill the usage worker

Merged
SA
2mo ago
+132-39

Problem

For a user opted into the leaderboard, usage.leaderboardPayload fails on every auto-publish attempt (the desktop checks every 5 minutes), so their usage never publishes and each attempt costs a dead worker thread plus ~110 s of a hung inline retry on the host-service main thread. Both issues come from one machine on release 1.25.1, and the sample events share a single trace: HOST-SERVICE-55 is the worker dying, HOST-SERVICE-56 is the inline retry of that same request timing out.

Reality check:

  1. User-visible harm: one opted-in user's leaderboard never updates, and their host-service burns a worker crash and a 110 s hung task every publish cycle.
  2. Reach: the events are from release 1.25.1 host-service (the desktop-bundled one). This change is in packages/host-service, so it ships with the next desktop/host-service release and reaches that build's successor on update.
  3. Instrument: code. Archiving leaves the user unpublishable and the 5-minute crash loop running. A Sentry-side filter is forbidden by repo law. There is no other layer: the reader is the only thing that touches the file.

Root cause

forEachLine in parse.ts used node:readline, which buffers a line until it sees a newline with no bound. A .jsonl under the transcript roots with a newline-free stretch longer than V8's maximum string (~512 MB) makes readline throw RangeError: Invalid string length inside the stream's data callback. That throw is not a rejection of the for await, so the surrounding try/catch never sees it:

  • In the host-worker thread it is an uncaught exception, so the worker dies (mechanism auto.child_process.worker_thread, HOST-SERVICE-55).
  • The pool then retries the task inline. On the main thread safety.ts swallows the same uncaught exception to keep the process alive, but the readline iteration never settles, so the task hits the 110 s timeout (HOST-SERVICE-56, (inline) in the message). The inline throws are absent from Sentry because the repeat throttle fingerprints them identically to the worker ones.

Reproduced end to end with a 640 MB newline-free fixture, running forEachLine at HEAD:

--- worker thread (as the host-worker pool runs it)
worker 'error' event after 355 ms: RangeError Invalid string length
worker exit code 1
--- inline (pool's crash retry on the main thread, safety.ts keeps the process alive)
uncaughtException after 262 ms: RangeError Invalid string length — staying up
inline forEachLine: still pending after 20s → would hit the 110s task timeout lines seen: 1

The 56 events are not a "large history exceeds the budget" timeout: no other machine times out, and every 56 event follows a 55 from the same request. That is why this PR does not add an incremental per-file aggregate cache; see the notes below.

Fix

  • forEachLine splits read chunks on \n itself (stripping a trailing \r) and never buffers more than MAX_LINE_LENGTH (32 MB). A longer line is skipped and reading resumes at the next newline, so no data file can throw from inside the worker. The cap is calibrated against real data: the longest line across 90 days of heavy Claude and Codex transcripts on a developer machine was 5.8 MB, and it is 16× below V8's limit.
  • The Antigravity collector (agy.ts) had its own readline over the same kind of file and now uses the shared reader, so the same walk has one bounded reader.
  • No new throw site, so no typed cause was added: nothing reads one.

Same fixture with the new reader, in a worker thread: { lines: [158, 158], ms: 261 }, exit code 0 (the two real entries around the 640 MB junk are delivered).

Negative test first: the new parse.test.ts case "skips a line past MAX_LINE_LENGTH" was run against the HEAD reader, which delivered the oversized line:

old readline reader → line lengths delivered: [ 5, 33554433, 4 ]
new bounded reader → line lengths delivered: [ 5, 4 ]

Verification

bunx biome check on the three changed files:

Checked 3 files in 10ms. No fixes applied.

cd packages/host-service && bun run typecheck:

$ tsc --noEmit --emitDeclarationOnly false

(no errors)

bun test src/trpc/router/usage:

 107 pass
 0 fail
 226 expect() calls
Ran 107 tests across 19 files. [282.00ms]

Full package suite, cd packages/host-service && bun run test, run twice. First run: 1647 pass, 8 todo, 1 fail (the failing test was not captured before the rerun). Second run:

 1648 pass
 8 todo
 0 fail
 3826 expect() calls
Ran 1656 tests across 159 files. [224.23s]

Out of scope, left as is

  • Incremental per-file aggregates (the task's part 2): not evidenced by these issues, and it needs an architecture decision because worker task handlers may not import the host DB (no-native-worker-imports ratchet), so persistence would have to be a separate file store or a prepare/result round-trip. Recommend a separate ticket if a real budget timeout ever shows up on a machine without a crash.
  • Every other reader in the walk (grok.ts, opencode.ts, cursor.ts readFile) surfaces failures as rejections, which the existing catches and allSettled already handle.

Refs HOST-SERVICE-55
Refs HOST-SERVICE-56

https://claude.ai/code/session_01BnLYGiunj6h6fGV1qRgk5b

Summary by cubic

Replaces the unbounded node:readline line reader in forEachLine with a chunk-splitting reader that skips any line past 32 MB, so one corrupt transcript file can no longer crash the usage worker with a RangeError: Invalid string length and hang the 110 s inline retry. Fixes HOST-SERVICE-55 and HOST-SERVICE-56.

  • The Antigravity collector now uses the same bounded reader instead of its own readline.
  • The cap is measured after stripping a trailing \r, so LF and CRLF files share one bound.
  • Added tests covering split multibyte characters, oversized lines, unterminated tails, CRLF boundaries, and unreadable files.
  • Real transcript lines top out around 5.8 MB, so the 32 MB cap only affects corrupt or foreign files.

Summary by CodeRabbit

  • Bug Fixes

    • Improved usage history processing for transcript files with varied line endings, incomplete final lines, and multibyte characters.
    • Prevented unusually large transcript lines from interrupting processing, allowing subsequent entries to continue being read.
    • Improved resilience when transcript files are unavailable or updated during processing.

  • Tests

    • Added coverage for line parsing, large entries, partial content, and unreadable files.