perf(pdf): bound OCR upload and rendered-page memory

Merged
CY
13d ago
+361-42

Summary

OCR-only PDF partitioning kept ten decoded page images in memory at once, and the OCR step copied the whole upload into one byte string for the entire OCR run. This PR renders one page at a time, copies uploads to temporary disk in 1 MiB blocks, and closes each rendered page image after OCR, including when OCR fails. Pixels, format, rotation metadata, page order, upload cursor behavior and caller-owned stream lifetime are preserved. Uploads may be any readable, seekable stream (duck-typed read/seek), not only BytesIO, SpooledTemporaryFile or buffered files.

Validation

  • New ownership, bounded-read, generator cancellation, OCR cleanup, native render parity and generic-stream tests pass locally; the original 13 also passed in an isolated Linux container.
  • Full PDF/OCR/utilities and new regression suite pass locally (with Korean OCR data installed).
  • OCR-only partitioning of rotated and content-bearing multi-page fixtures in a Linux container: every element matches the baseline JSON hashes exactly.
  • Large synthetic OCR-only PDF in a memory-limited Linux container: main runs out of memory where this branch completes at the same limit. With enough memory for both, peak RSS drops to under a third, runtime is unchanged and output hashes match.
  • Uploads opened through a bytes path are re-read from that path, leaving the caller's cursor untouched (regression test for text and binary modes).
  • Repository Ruff checks/format and the Linux version-sync checker pass.

Operational impact

OCR rendering reads uploads from temporary disk instead of holding a full upload-sized copy in memory. Spooled uploads (SpooledTemporaryFile) are still copied in full, briefly, by spooled_to_bytes_io_if_needed for the text-extraction check before the OCR branch; the saving is that this copy is not held while OCR runs. There is no new upload size cap. Working render memory is bounded by one page; extracted output and memory retained by callers still scale with content. Public render-generator images remain usable after iteration and are owned by the caller.

Follow-up

_pdf_source_path and convert_to_bytes (#4419) both decide how to read a given upload type. Once #4419 lands, both should use one shared helper in unstructured/partition/common/common.py that copies a stream to disk in blocks.

Release

Targets 0.27.32 (0.27.31 was taken by #4538, which merged first.) Released together with #4419, #4516, #4517, #4524, #4523, #4562 (XLSX workbook reader) and #4563 (dense worksheets). Each PR takes the next 0.27.32-devN in that merge order (0.27.32-dev1 here); whichever merges after another rebases and renumbers the CHANGELOG heading, as usual for this repo.