perf(pdf): vectorize aggregate_embedded_text_by_block's IoU and drop redundant slicing
Summary
aggregate_embedded_text_by_block() runs once per layout region that needs text, from merge_out_layout_with_ocr_layout() (OCR) and merge_inferred_with_extracted_layout() (hi_res). Per call it:
- tested
if sum(mask):— builtinsumiterates a numpy bool array element by element; nowmask.any(); - sliced
source_regionswith the same mask three times; now once; - computed
_aggregated_iouwith a Python loop over the selected boxes; now vectorized with numpy.
Numerical equivalence
_aggregated_iou accumulates with np.cumsum, which adds left to right like the old loop (np.sum's pairwise summation could differ in the last bit), and seeds the sums with the coordinates' own dtype, so float32 input still accumulates in float32 as the loop did. Results are bit-identical, with the same return dtype, for both float32 and float64.
test_aggregated_iou_matches_scalar_loop_bit_for_bit pins this: it compares against the old scalar loop over 500 random cases per dtype, including zero-size and inverted boxes, and requires exact equality. (In production, TextRegions converts coordinates to float64, so the float32 case isn't currently reachable; the test keeps the claim unconditional.)
Performance
Walltime, pytest-codspeed run locally (Apple Silicon), merge_out_layout_with_ocr_layout with 60 regions and 1,500 OCR words on one page:
| before | after | |
|---|---|---|
| per page | 2.7 ms | 0.39 ms |
Validation
test_unstructured/partition/pdf_image/test_pdfminer_processing.py,test_ocr.py,test_pdf.pyandtest_merge_elements.pypass.