perf(text_type): run the non-alpha check before the spaCy-backed cap-ratio check

KR
6h ago
+23-6

Summary

is_possible_narrative_text() called exceeds_cap_ratio() before under_non_alpha_ratio(). exceeds_cap_ratio() calls sentence_count() and word_tokenize(), which run the spaCy pipeline; under_non_alpha_ratio() is a single character scan. Both checks only ever return False, so running the cheap one first gives the same result and skips spaCy for text like ----------, $ 1,234.56 $ 7,890.12 or *** === +++.

is_possible_title() already orders its checks this way.

A new test, test_is_possible_narrative_text_rejects_low_alpha_text_without_nlp, makes word_tokenize, pos_tag and sent_tokenize raise and asserts low-alpha text is still rejected, so the optimization can't silently regress. It fails on the previous check order.

Behavior notes for maintainers

With valid configuration the classification result is unchanged. Three observable side effects change, all only for text that fails the non-alpha check:

  • Trace logging: text that fails both checks no longer emits the "exceeds cap ratio" detail line, nor the per-sentence detail lines from sentence_count().
  • Malformed threshold env vars: UNSTRUCTURED_NARRATIVE_TEXT_NON_ALPHA_THRESHOLD is now read first, so a malformed value raises ValueError even for text the cap-ratio check would have rejected first (e.g. "RISK FACTORS"). Conversely, a malformed UNSTRUCTURED_NARRATIVE_TEXT_CAP_THRESHOLD is no longer read for low-alpha text. On main either malformed value already raises for most text.
  • spaCy failures: if the spaCy model can't be loaded, low-alpha text is now rejected without reaching it instead of raising.

Performance

Walltime, pytest-codspeed run locally (Apple Silicon), 120 strings:

beforeafter
is_possible_narrative_text167.7 ms84.0 ms

This is a worst case: the benchmark clears the spaCy caches every round, so documents with repeated strings gain less. The gain also depends on how much of a document is symbol- or number-heavy; 5 of the 12 sample strings were. Prose-only input gains little.

Validation

  • test_unstructured/partition/test_text_type.py passes (119 tests).