perf: Free decode buffers sooner when restoring original elements

Merged
CY
12d ago
+213-1

Summary

Restoring orig_elements now releases each intermediate buffer before the next stage builds its own copy. Previously the compressed bytes, decompressed bytes and JSON text all stayed alive until the end.

Restoring one element that carries a very large image payload:

  • Peak memory drops by roughly a quarter.
  • Under a constrained memory limit, main runs out of memory and this branch completes.
  • Restored elements are identical.

Behavior

  • Unchanged:
    • the Base64, zlib, UTF-8, JSON and element constructors;
    • restored values, metadata order and error behavior;
    • the existing 200 MiB decompressed-byte cap.
  • No new cap is added. The existing cap bounds UTF-8 bytes, not Python string size, so wide-Unicode JSON can still grow in memory. The full element graph is still returned.
  • This removes avoidable overlap between buffers. It does not guarantee that every input fits a fixed memory budget.

Validation

  • Staging tests pass. New regression tests cover Unicode, nested originals, metadata and order, malformed input, trailing data and the exact size boundary.
  • A tracemalloc test checks that json.loads and element construction each start with only their own input alive. Removing any one of the three releases fails it.
  • The new tests also pass in an isolated Linux container against the module from this commit (import path and SHA256 verified).
  • Included benchmark: one element with a very large BMP stored as a Base64 payload.
    • Under a constrained memory limit, main runs out of memory. This branch completes with identical metadata, text and payload hash.
    • With enough memory for both, peak RSS falls as above; decode time is essentially unchanged.
    • Escaped wide-Unicode text and raw UTF-8 mixed with the image payload also restore identically, with a similar peak-memory reduction.
  • Ruff, diff checks and the version checker pass.

Release

Targets 0.27.32 (0.27.31 was taken by #4538, which merged first.) Released together with #4419, #4516, #4517, #4524, #4523, #4562 (XLSX workbook reader) and #4563 (dense worksheets). Each PR takes the next 0.27.32-devN in that merge order (0.27.32-dev3 here); whichever merges after another rebases and renumbers the CHANGELOG heading, as usual for this repo.

#4517 changes the serializing side of orig_elements in the same module. A test merge with this branch conflicts only on version and changelog lines, passes the staging, element and chunking suites, and round-trips identical elements.