perf(chunking): bound image payload and original-element serialization memory

Merged
CY
13d ago
+646-15

Summary

A stream of empty images can hold every base64 payload until a later text element even when include_orig_elements=False and those image fields are dropped from output. Fold metadata incrementally for contiguous empty-image runs and release excluded image payloads immediately, preserving metadata order and chunk boundaries.

Original-element serialization now adjusts and encodes one element at a time, escapes large JSON strings in bounded fragments (values without large strings still use one standard json.dumps call), and spills compressed buffers above 1 MiB to temporary disk. Avoid the redundant deep copy of the whole original-element graph in ElementMetadata.to_dict().

Validation

  • Chunking, original-element staging and element suites pass locally; the new regression cases also pass in an isolated Linux container.
  • 120 mixed-image/text/title/table/page-break cases with overlap preserve chunk boundaries and metadata against the identity-preserving path. Original Image identity remains intact with the default include_orig_elements=True.
  • Exact legacy JSON field order is preserved, including continuation metadata and empty originals.
  • Exact legacy compressed bytes match for empty inputs, multiple elements, long Unicode/control-character strings, metadata precision and sorted JSON keys on the same zlib implementation.
  • Many streamed large image payloads in a Linux container: chunking peak RSS drops to under a quarter and runs faster; serialization peak RSS drops by roughly an order of magnitude with slightly lower runtime. Output hashes match exactly.
  • Under a constrained container memory limit, both main cases run out of memory; both cases on this branch complete with no OOM events.
  • Metadata nested as deeply as the standard JSON encoder accepts still serializes: the large-string scan is iterative, not recursive (regression test with 600 nested lists).
  • Repository Ruff checks/format and the Linux version-sync checker pass.

Operational impact

Compressed original-element buffers above 1 MiB use temporary disk. Final serialized output and metadata actually included in chunks still require memory proportional to output. include_orig_elements=True continues retaining original objects; changing that identity contract would require a separate API decision.

Release

Targets 0.27.32 (0.27.31 was taken by #4538, which merged first.) Released together with #4419, #4516, #4517, #4524, #4523, #4562 (XLSX workbook reader) and #4563 (dense worksheets). Each PR takes the next 0.27.32-devN in that merge order (0.27.32-dev2 here); whichever merges after another rebases and renumbers the CHANGELOG heading, as usual for this repo.