perf(text_type): run the non-alpha check before the spaCy-backed cap-ratio check
Summary
is_possible_narrative_text() called exceeds_cap_ratio() before under_non_alpha_ratio(). exceeds_cap_ratio() calls sentence_count() and word_tokenize(), which run the spaCy pipeline; under_non_alpha_ratio() is a single character scan. Both checks only ever return False, so running the cheap one first gives the same result and skips spaCy for text like ----------, $ 1,234.56 $ 7,890.12 or *** === +++.
is_possible_title() already orders its checks this way.
A new test, test_is_possible_narrative_text_rejects_low_alpha_text_without_nlp, makes word_tokenize, pos_tag and sent_tokenize raise and asserts low-alpha text is still rejected, so the optimization can't silently regress. It fails on the previous check order.
Behavior notes for maintainers
With valid configuration the classification result is unchanged. Three observable side effects change, all only for text that fails the non-alpha check:
- Trace logging: text that fails both checks no longer emits the "exceeds cap ratio" detail line, nor the per-sentence detail lines from
sentence_count(). - Malformed threshold env vars:
UNSTRUCTURED_NARRATIVE_TEXT_NON_ALPHA_THRESHOLDis now read first, so a malformed value raisesValueErroreven for text the cap-ratio check would have rejected first (e.g."RISK FACTORS"). Conversely, a malformedUNSTRUCTURED_NARRATIVE_TEXT_CAP_THRESHOLDis no longer read for low-alpha text. Onmaineither malformed value already raises for most text. - spaCy failures: if the spaCy model can't be loaded, low-alpha text is now rejected without reaching it instead of raising.
Performance
Walltime, pytest-codspeed run locally (Apple Silicon), 120 strings:
| before | after | |
|---|---|---|
is_possible_narrative_text | 167.7 ms | 84.0 ms |
This is a worst case: the benchmark clears the spaCy caches every round, so documents with repeated strings gain less. The gain also depends on how much of a document is symbol- or number-heavy; 5 of the 12 sample strings were. Prose-only input gains little.
Validation
test_unstructured/partition/test_text_type.pypasses (119 tests).