perf: Free image buffers sooner during figure extraction

Merged
CY
13d ago
+301-32

Summary

Image and PDF figure extraction now frees each decoded page, crop and JPEG buffer as soon as it is no longer needed. Image uploads are copied to the temporary file in 1 MiB blocks instead of being held in memory whole.

Extracting full-page figures from a very large synthetic image:

  • Peak memory drops by roughly 40%.
  • Under a constrained memory limit, main runs out of memory and this branch completes.
  • Output is byte-identical.

What changed

  • Uploads are copied to the existing temporary file in 1 MiB blocks.
  • Decoded pages, RGBA intermediates, output crops and JPEG buffers are closed as soon as their pixels or bytes are used.

Behavior

  • Unchanged: JPEG bytes, Base64 payloads, saved filenames, element order, metadata and padding. Caller-owned streams keep the same rewind/EOF behavior.
  • No size cap or disk policy is added; image uploads already used the same temporary file.
  • Output payloads, and the current page plus its crop, can still be large.

Validation

  • Extraction/utility tests pass. They include:
    • exact JPEG bytes for RGB, RGBA and grayscale, in both payload and saved-file output;
    • bounded upload reads;
    • caller stream ownership and EOF;
    • multi-crop cleanup after an encoder failure.
  • The new regression tests also pass in an isolated Linux container against the module from this commit (import path and SHA256 verified).
  • Included benchmark (scripts/performance/benchmark_image_extraction_memory.py): full-page figures from a deterministic very-high-resolution image.
    • Under a constrained memory limit, main runs out of memory and this branch completes.
    • A healthy main run produces the same metadata and payload hash.
    • With enough memory for both, peak RSS falls as above and extraction is slightly faster.
    • This measures the extraction stage, not end-to-end partition latency.
  • Full native PDF/image suite passes. Ruff, diff checks and the version checker pass.

Release

Targets 0.27.32 (0.27.31 was taken by #4538, which merged first.) Released together with #4419, #4516, #4517, #4524, #4523, #4562 (XLSX workbook reader) and #4563 (dense worksheets). Each PR takes the next 0.27.32-devN in that merge order (0.27.32-dev4 here); whichever merges after another rebases and renumbers the CHANGELOG heading, as usual for this repo.

#4516 also edits pdf_image_utils.py, but a different function. Downstream dependency bumps need the published release.