perf(xlsx): reduce dense-worksheet partitioning memory

Merged
CY
13d ago
+370-68

What

partition_xlsx() with subtable detection (the default) needs substantially less memory on dense worksheets, and the saving grows with sheet size. Output is unchanged.

On synthetic dense sheets (a ten-column sheet split into three dense subtables, half integers, half short Unicode strings), peak RSS in a fresh process drops by roughly 40–60% depending on size, with comparable runtime. Working memory per populated cell drops to roughly a third. Serialized element JSON is byte-identical before and after.

Why

Two whole-sheet intermediates dominated memory on dense sheets:

  1. Subtable detection built a NetworkX graph with a node per populated cell and an edge per neighbor pair. It now uses an iterative four-neighbor flood fill over populated cells (the approach from #4357), with the same component admission order, cell membership, row-overlap merging, table order and bounding boxes.
  2. Table HTML was rendered for the whole table at once (pandas to_html string, its parsed DOM and a full-table token list). Rows are now formatted by pandas' formatter for the whole column, so float/date/string formatting is unchanged, and each row is rendered and parsed on its own before the next.

networkx is no longer needed at runtime by the XLSX partitioner, so it moves from the xlsx extra to the test dependencies (it's still used by other extras that depend on it).

Behavior

  • Same elements, text, text_as_html and metadata; exact-output tests cover mixed types, NaN/empty cells, wide floats that switch to scientific notation in another row, dates and HTML escaping.
  • The renderer uses pandas' private formatter classes, as the existing reader already does for OpenpyxlReader. Verified on pandas 2.1.4 (renderer-parity comparisons), 2.2.3 and 2.3.3.
  • The worksheet cell limit, empty-row compaction, empty-sheet handling and XLS padding handling are unchanged.

Validation

  • XLSX and dense-memory suites pass. New tests compare component detection and rendered HTML against the previous implementation on generated sheets.
  • Ruff check/format pass.

Release

Sets the release to 0.27.32 (0.27.31 was taken by #4538, which merged first). It merges after #4419, #4516, #4517 and #4562, and its CHANGELOG collects their entries and this one into a single 0.27.32 section. #4524 and #4523 are not included and will go into a later release.

Supersedes #4357.