fix: Hindi and Bengali paragraphs misclassified as Title
Problem
Text in Hindi, Bengali and other Indic scripts is misclassified in both directions:
-
Paragraphs become
Title.partition_texton six two-sentence Hindi paragraphs returns 3Titles, and on three Bengali paragraphs returns 2Titles.chunk_by_titlethen starts a new chunk at each false title:Document Before After Hindi (6 paragraphs) 3/6 Title, 4 chunks0/6 Title, 1 chunkBengali (3 paragraphs) 2/3 Title, 3 chunks0/3 Title, 1 chunkEnglish control (3 paragraphs) 0/3 Title, 1 chunk0/3 Title, 1 chunk (unchanged)(
chunk_by_title(elements, max_characters=2000, combine_text_under_n_chars=0)) -
Real headings can't be detected as
Title.is_possible_title("हिंदी")("Hindi") returnsFalse, becauseunder_non_alpha_ratio("हिंदी")says the word is mostly non-letters.
Root causes
- Vowel signs aren't letters to
str.isalpha(). In Devanagari and Bengali, vowel signs (matras), viramas and anusvaras are Unicode combining marks (categoryM*), so"हिंदी".isalpha()isFalse: only 2 of its 5 code points are letters.under_non_alpha_ratiocounts the marks as non-letters, so short words like "हिंदी" fall under the 0.5 threshold.exceeds_cap_ratiokeeps only tokens wheretk.isalpha()is true. In a Hindi sentence where every word carries a vowel sign, nothing is left, and the "no tokens" branch returnsTrue("too many capitals"). The paragraph then fails the narrative check.
- The danda (
।,॥) isn't a reliable sentence end. The English spaCy model sometimes splits at।and sometimes doesn't. When it doesn't, a two-sentence paragraph counts as one sentence. That sends it into the single-sentence branch ofexceeds_cap_ratioabove, and into the English verb check.
Fix
text_type.py: a character counts as a letter if it is alphabetic or a combining mark.under_non_alpha_ratio: counts marks as letters, written inline for speed (char.isalpha() or unicodedata.category(char)[0] == "M").exceeds_cap_ratio: a token counts as a word if it has at least one letter and every character is a letter or a mark (_is_word), instead oftk.isalpha().
nlp/tokenize.py: after spaCy's sentence segmentation, each sentence is split again on(?<=[।॥])\s+. This is inside the existinglru_cached_tokenize_for_cache, so caching is unchanged andsent_tokenize/sentence_countpick it up.
Unchanged: thresholds, the English verb check, is_possible_narrative_text/is_possible_title logic, and behavior for text without combining marks or dandas. All existing tests in the touched files pass.
Tests
New tests in test_text_type.py, nlp/test_tokenize.py and partition/test_text.py:
under_non_alpha_ratio("हिंदी")isFalse;is_possible_title("हिंदी")isTrueexceeds_cap_ratioisFalsefor "हिंदी सबसे अधिक बोली जाने वाली भाषा है।" (every word has a vowel sign)sentence_countof "हिंदी सबसे अधिक बोली जाने वाली भाषा है। गंगा पवित्र नदी है।" is 2 (was 1);sent_tokenizesplits on।and॥is_possible_narrative_textisTruefor each Hindi/Bengali paragraph;partition_textreturns onlyNarrativeTextfor both documents- Still works:
under_non_alpha_ratioon full paragraphs,is_possible_title("परिचय"), and the existing English title/narrative cases
12 of the new tests fail on main and pass with this change; the rest guard behavior that already worked. test_text_type.py, nlp/test_tokenize.py and partition/test_text.py: 246 passed. ruff check and ruff format --check pass on the changed files.
To reproduce:
from unstructured.partition.text import partition_text
paras = ["दिल्ली में सरकार की बैठकें होती हैं। संसद भी यहीं है।",
"কলকাতা পশ্চিমবঙ্গের রাজধানী। এখানে অনেক মানুষ থাকে।"]
print([type(e).__name__ for e in partition_text(text="\n\n".join(paras))])
# main: ['Title', 'Title'] this PR: ['NarrativeText', 'NarrativeText']
Performance
10,000 calls each, best of 5, Apple M2, Python 3.13:
| Text | Function | main | This PR |
|---|---|---|---|
| English paragraph | under_non_alpha_ratio | 8.20 µs | 8.26 µs |
| English paragraph | exceeds_cap_ratio | 6.47 µs | 6.00 µs |
| Hindi paragraph | under_non_alpha_ratio | 3.15 µs | 4.16 µs |
| Hindi paragraph | exceeds_cap_ratio | 5.98 µs | 5.90 µs |
English is within noise. Hindi under_non_alpha_ratio does about 1 µs more work because it now looks up the category of each mark. The exceeds_cap_ratio numbers are the cached path; classifying a new element is dominated by the spaCy run (milliseconds), which this PR doesn't change.
Known limitation (possible follow-up)
A single-sentence Hindi/Bengali paragraph can still become a Title, e.g. "दिल्ली में सरकार की बैठकें होती हैं।", "भारत एक विशाल देश है।" or "রবীন্দ্রনাথ ঠাকুর নোবেল পুরস্কার পেয়েছিলেন।". Both character checks now pass, but with fewer than 2 sentences is_possible_narrative_text runs the English-only verb check (contains_verb), whose result for Hindi depends on spaCy happening to tag some word as an English verb.
partition_text detects hin (it's in element.metadata.languages) but doesn't pass languages to is_possible_narrative_text; is_possible_narrative_text(t, languages=["hin"]) returns True. I'm happy to send that as a separate PR.
Overlap with #4679
#4679 reorders the under_non_alpha_ratio and exceeds_cap_ratio calls inside is_possible_narrative_text. This PR only changes the bodies of those two functions (plus new helpers above is_possible_narrative_text), so the two shouldn't conflict, and the reorder doesn't change any result here since both checks can only make the function return False early.