perf(chunking): normalize each table cell once when splitting tables (about 1.7x faster)
Summary
Splitting a table during chunking normalizes the text of every cell several times. _RowAccumulator asks each HtmlRow for its cell texts in will_fit(), again in add_rows(), again in flush() when it builds the chunk text, and once more when a row is retried after a flush. Each call ran normalize_html_cell_text() on every cell, and that is most of the cost of splitting a large table.
HtmlRow.iter_cell_texts() now reads a cached_property that normalizes the row's cells once, the same way HtmlRow.html and HtmlRow.text_len are already cached. Nothing changes a row's <tr> after the HtmlRow is built (_clipped_tr() works on a deep copy), so the cached texts can't go stale. Chunk output is unchanged.
| Case (chunking only) | before | after | speedup |
|---|---|---|---|
CSV, 5 rows x 4 cols, max_characters=500 | 0.74 ms | 0.73 ms | 1.0x |
CSV, 30 rows x 4 cols, max_characters=500 | 3.8 ms | 2.9 ms | 1.3x |
CSV, 500 rows x 12 cols, max_characters=1000 | 88 ms | 49 ms | 1.8x |
CSV, 3,000 rows x 6 cols, max_characters=500, chunk_elements() | 301 ms | 193 ms | 1.6x |
HTML report, 20 sections each with a 30-row table, max_characters=1500 | 131 ms | 95 ms | 1.4x |
CPU time, median of 5 alternating same-process rounds, Python 3.13, lxml 6.1.3, Ryzen 5 5600X, Linux. Token-based chunking (max_tokens=100, 1,000-row CSV) went from 146 ms to 108 ms.
Validation
- Old and new
iter_cell_texts()give identicalchunk_by_title()andchunk_elements()output (type, text andtext_as_html) and identical per-row cell texts on randomly generated tables with non-ASCII text, entities,<br>, inline markup, nested tables, blank cells,<th>,colspan/rowspan(including0and invalid values),<thead>/<tbody>, character and token limits, overlap,repeat_table_headersandisolate_table. - A new test checks that iterating a row's cell texts three times normalizes each cell once.
test_unstructured/chunking/andtest_unstructured/common/pass;ruff checkandruff format --checkare clean.
Checklist
- The PR title is a user-facing release-note entry with a conventional commit prefix.
- Apply a category label (
enhancement,bug, or an appropriate existing label), or ask a maintainer to apply it. - For breaking changes, use
!in the title and explain impact and migration above.