fix(parsers): yield the event loop during PDF extraction and CSV row coercion

Merged
WA
19d ago
+174-23

Summary

  • PDF extraction: pdf.js runs its fake worker on the request thread and resolves every message as a microtask, so neither the page loop nor a page's streamTextContent read loop ever let timers or I/O run. One shared time-sliced check (createEventLoopSlicer, setImmediate from node:timers/promises) now yields a macrotask once 10ms have passed since the last yield, in the page loop, inside the stream read loop (covers one page with a huge content stream), and in page assembly, between its whole-document passes and per page (replaces the fixed sleep(0) every 32 pages).
  • Table CSV append/replace: rows are coerced in slices of 2,000 with a yield between slices (coerceRowsForTable is a 1:1 map, so rows and cellsRejected are identical).
  • Row limit is now checked right after parsing for both append and replace, before coercion and before claiming the job slot.

User-visible changes:

  • An over-limit replace now fails earlier with a validation message ("Replace would exceed table row limit (N). The file has X rows, Y over.", still HTTP 400) instead of the in-transaction row-limit error, and without claiming the job slot.
  • An over-limit file now reports the row-limit validation error before a concurrent-import conflict (previously the slot claim ran first).
  • PDF extraction output and CSV coercion output are unchanged (hashes below).

Type of Change

  • Bug fix

Testing

  • New regression tests, each proven red on pre-fix code:
    • pdf-parser-cancellation.test.ts: a timer runs between CPU-bound pages; a timer runs while one page streams many chunks (red with only the inner read-loop yield removed)
    • import.test.ts: a timer runs during coercion of a 50,000-row file before the write; an over-limit append/replace returns the row-limit validation error while the slot is held by another job (a within-limit replace still gets conflict)
  • Ran pdf-parser-cancellation, pdf-parser, pdf-parser-structure, import tests (32 passed), bun run lint. Full type-check/audits/test suite left to CI.
  • Local before/after (generated fixtures; max gap between 1ms interval ticks while the call runs, 3 runs):
FixtureBefore max blockAfter max blockWall timeOutput hash
400-page text PDF667–701ms47–51ms~720ms bothidentical (normal + complete mode)
1 page, 30,000 text ops258–273ms35–42ms~270ms bothidentical (normal + complete mode)
23 MB CSV, 5 typed columns, I/O-paced 64KB chunks265ms74ms~1.05s bothcoerced rows + cellsRejected identical

The remaining PDF block is a single whole-document assembly pass (furniture suppression or word collection); the CSV residual includes the bench's own hashing of the rows.

Checklist

  • Code follows project style guidelines
  • Self-reviewed my changes
  • Tests added/updated and passing (new tests pass the test-audit authoring gate)
  • No new warnings introduced
  • I confirm that I have read and agree to the terms outlined in the Contributor License Agreement (CLA)