fix(extractor): keep word spaces painted as squeezed whitespace runs

Merged
AB
2mo ago
+371-22

Summary

Some producers emit one text operator per word and paint the separating space as its own whitespace-only run, squeezed with a negative Tc until the gap it leaves is narrower than the word-space thresholds. Whitespace runs never became items, so the words merged (forthe, stratifiedby, Bankassets). Honoring Tz in #510 moved more of these gaps under the threshold, which is how it surfaced.

Change

  • PendingSpace in the content-stream parser: a whitespace-only run narrower than 0.2 em that starts where the previous item ends is held; when the next run arrives, the previous item gets a trailing space only if both neighbours are contiguous, same-size (±15%), left-to-right alphanumeric text. Superscripts, signs, joining punctuation and RTL runs keep their current handling.
  • Applied to Tj and whitespace-only TJ arrays in both the page parser and the Form XObject parser.
  • merge_text_items no longer adds a second space after text that already ends in whitespace (also collapses NBSP/tab + space doubles).
  • RunGeometry::is_upright shared with the item-level rule.

Tests

  • Unit: squeezed runs give the space (Tj, TJ, 99 Tz, repeated runs, digits); wide runs are left to gap detection; runs away from their neighbours, rotated runs, and runs with no prior item are dropped; superscript, punctuation and sign junctions are unchanged; merge loop does not double an explicit trailing space.
  • Integration: form-parity case under Tz in tests/horizontal_scale.rs.

Eval footprint

8 snapshots change in the regression corpus: two documents gain the missing word spaces (20 junctions), four collapse a doubled NBSP/tab + space, one symbolic-font document reorders garbage. Two broader variants (attaching the space unconditionally, and gating on run width alone) were measured first and rejected for regressing superscripts, dot leaders, page-number stripping, sign/number junctions and RTL text.

Summary by cubic

Fixes words being merged when producers paint the separating space as its own whitespace-only run squeezed with negative character spacing, so text like forthe now extracts as for the.

  • Adds a PendingSpace state that holds a squeezed whitespace run and, when the next run arrives, gives the previous item a trailing space if both neighbors are contiguous, same-size, left-to-right alphanumeric text.
  • Applies to Tj and whitespace-only TJ arrays in both the page parser and the Form XObject parser, and respects rotation and baseline shifts.
  • Prevents merge_text_items from adding a second space after text that already ends in whitespace, which also collapses NBSP/tab + space doubles.