perf(pdf): vectorize aggregate_embedded_text_by_block's IoU and drop redundant slicing

KR
5h ago
+49-13

Summary

aggregate_embedded_text_by_block() runs once per layout region that needs text, from merge_out_layout_with_ocr_layout() (OCR) and merge_inferred_with_extracted_layout() (hi_res). Per call it:

  • tested if sum(mask): — builtin sum iterates a numpy bool array element by element; now mask.any();
  • sliced source_regions with the same mask three times; now once;
  • computed _aggregated_iou with a Python loop over the selected boxes; now vectorized with numpy.

Numerical equivalence

_aggregated_iou accumulates with np.cumsum, which adds left to right like the old loop (np.sum's pairwise summation could differ in the last bit), and seeds the sums with the coordinates' own dtype, so float32 input still accumulates in float32 as the loop did. Results are bit-identical, with the same return dtype, for both float32 and float64.

test_aggregated_iou_matches_scalar_loop_bit_for_bit pins this: it compares against the old scalar loop over 500 random cases per dtype, including zero-size and inverted boxes, and requires exact equality. (In production, TextRegions converts coordinates to float64, so the float32 case isn't currently reachable; the test keeps the claim unconditional.)

Performance

Walltime, pytest-codspeed run locally (Apple Silicon), merge_out_layout_with_ocr_layout with 60 regions and 1,500 OCR words on one page:

beforeafter
per page2.7 ms0.39 ms

Validation

  • test_unstructured/partition/pdf_image/test_pdfminer_processing.py, test_ocr.py, test_pdf.py and test_merge_elements.py pass.