fix(parsers): yield the event loop during PDF extraction and CSV row coercion
Merged
WA
19d ago+174+174+012345678901234567890123456789-23-23-01234567890123456789
Summary
- PDF extraction: pdf.js runs its fake worker on the request thread and resolves every message as a microtask, so neither the page loop nor a page's
streamTextContentread loop ever let timers or I/O run. One shared time-sliced check (createEventLoopSlicer,setImmediatefromnode:timers/promises) now yields a macrotask once 10ms have passed since the last yield, in the page loop, inside the stream read loop (covers one page with a huge content stream), and in page assembly, between its whole-document passes and per page (replaces the fixedsleep(0)every 32 pages). - Table CSV append/replace: rows are coerced in slices of 2,000 with a yield between slices (
coerceRowsForTableis a 1:1 map, so rows andcellsRejectedare identical). - Row limit is now checked right after parsing for both append and replace, before coercion and before claiming the job slot.
User-visible changes:
- An over-limit replace now fails earlier with a validation message ("Replace would exceed table row limit (N). The file has X rows, Y over.", still HTTP 400) instead of the in-transaction row-limit error, and without claiming the job slot.
- An over-limit file now reports the row-limit validation error before a concurrent-import conflict (previously the slot claim ran first).
- PDF extraction output and CSV coercion output are unchanged (hashes below).
Type of Change
- Bug fix
Testing
- New regression tests, each proven red on pre-fix code:
pdf-parser-cancellation.test.ts: a timer runs between CPU-bound pages; a timer runs while one page streams many chunks (red with only the inner read-loop yield removed)import.test.ts: a timer runs during coercion of a 50,000-row file before the write; an over-limit append/replace returns the row-limit validation error while the slot is held by another job (a within-limit replace still getsconflict)
- Ran
pdf-parser-cancellation,pdf-parser,pdf-parser-structure,importtests (32 passed),bun run lint. Full type-check/audits/test suite left to CI. - Local before/after (generated fixtures; max gap between 1ms interval ticks while the call runs, 3 runs):
| Fixture | Before max block | After max block | Wall time | Output hash |
|---|---|---|---|---|
| 400-page text PDF | 667–701ms | 47–51ms | ~720ms both | identical (normal + complete mode) |
| 1 page, 30,000 text ops | 258–273ms | 35–42ms | ~270ms both | identical (normal + complete mode) |
| 23 MB CSV, 5 typed columns, I/O-paced 64KB chunks | 265ms | 74ms | ~1.05s both | coerced rows + cellsRejected identical |
The remaining PDF block is a single whole-document assembly pass (furniture suppression or word collection); the CSV residual includes the bench's own hashing of the rows.
Checklist
- Code follows project style guidelines
- Self-reviewed my changes
- Tests added/updated and passing (new tests pass the
test-auditauthoring gate) - No new warnings introduced
- I confirm that I have read and agree to the terms outlined in the Contributor License Agreement (CLA)