fix: Hindi and Bengali paragraphs misclassified as Title

SH
5d ago
+168-8

Problem

Text in Hindi, Bengali and other Indic scripts is misclassified in both directions:

  1. Paragraphs become Title. partition_text on six two-sentence Hindi paragraphs returns 3 Titles, and on three Bengali paragraphs returns 2 Titles. chunk_by_title then starts a new chunk at each false title:

    DocumentBeforeAfter
    Hindi (6 paragraphs)3/6 Title, 4 chunks0/6 Title, 1 chunk
    Bengali (3 paragraphs)2/3 Title, 3 chunks0/3 Title, 1 chunk
    English control (3 paragraphs)0/3 Title, 1 chunk0/3 Title, 1 chunk (unchanged)

    (chunk_by_title(elements, max_characters=2000, combine_text_under_n_chars=0))

  2. Real headings can't be detected as Title. is_possible_title("हिंदी") ("Hindi") returns False, because under_non_alpha_ratio("हिंदी") says the word is mostly non-letters.

Root causes

  1. Vowel signs aren't letters to str.isalpha(). In Devanagari and Bengali, vowel signs (matras), viramas and anusvaras are Unicode combining marks (category M*), so "हिंदी".isalpha() is False: only 2 of its 5 code points are letters.
    • under_non_alpha_ratio counts the marks as non-letters, so short words like "हिंदी" fall under the 0.5 threshold.
    • exceeds_cap_ratio keeps only tokens where tk.isalpha() is true. In a Hindi sentence where every word carries a vowel sign, nothing is left, and the "no tokens" branch returns True ("too many capitals"). The paragraph then fails the narrative check.
  2. The danda (।, ॥) isn't a reliable sentence end. The English spaCy model sometimes splits at । and sometimes doesn't. When it doesn't, a two-sentence paragraph counts as one sentence. That sends it into the single-sentence branch of exceeds_cap_ratio above, and into the English verb check.

Fix

  • text_type.py: a character counts as a letter if it is alphabetic or a combining mark.
    • under_non_alpha_ratio: counts marks as letters, written inline for speed (char.isalpha() or unicodedata.category(char)[0] == "M").
    • exceeds_cap_ratio: a token counts as a word if it has at least one letter and every character is a letter or a mark (_is_word), instead of tk.isalpha().
  • nlp/tokenize.py: after spaCy's sentence segmentation, each sentence is split again on (?<=[।॥])\s+. This is inside the existing lru_cached _tokenize_for_cache, so caching is unchanged and sent_tokenize/sentence_count pick it up.

Unchanged: thresholds, the English verb check, is_possible_narrative_text/is_possible_title logic, and behavior for text without combining marks or dandas. All existing tests in the touched files pass.

Tests

New tests in test_text_type.py, nlp/test_tokenize.py and partition/test_text.py:

  • under_non_alpha_ratio("हिंदी") is False; is_possible_title("हिंदी") is True
  • exceeds_cap_ratio is False for "हिंदी सबसे अधिक बोली जाने वाली भाषा है।" (every word has a vowel sign)
  • sentence_count of "हिंदी सबसे अधिक बोली जाने वाली भाषा है। गंगा पवित्र नदी है।" is 2 (was 1); sent_tokenize splits on । and ॥
  • is_possible_narrative_text is True for each Hindi/Bengali paragraph; partition_text returns only NarrativeText for both documents
  • Still works: under_non_alpha_ratio on full paragraphs, is_possible_title("परिचय"), and the existing English title/narrative cases

12 of the new tests fail on main and pass with this change; the rest guard behavior that already worked. test_text_type.py, nlp/test_tokenize.py and partition/test_text.py: 246 passed. ruff check and ruff format --check pass on the changed files.

To reproduce:

from unstructured.partition.text import partition_text
paras = ["दिल्ली में सरकार की बैठकें होती हैं। संसद भी यहीं है।",
         "কলকাতা পশ্চিমবঙ্গের রাজধানী। এখানে অনেক মানুষ থাকে।"]
print([type(e).__name__ for e in partition_text(text="\n\n".join(paras))])
# main: ['Title', 'Title']   this PR: ['NarrativeText', 'NarrativeText']

Performance

10,000 calls each, best of 5, Apple M2, Python 3.13:

TextFunctionmainThis PR
English paragraphunder_non_alpha_ratio8.20 µs8.26 µs
English paragraphexceeds_cap_ratio6.47 µs6.00 µs
Hindi paragraphunder_non_alpha_ratio3.15 µs4.16 µs
Hindi paragraphexceeds_cap_ratio5.98 µs5.90 µs

English is within noise. Hindi under_non_alpha_ratio does about 1 µs more work because it now looks up the category of each mark. The exceeds_cap_ratio numbers are the cached path; classifying a new element is dominated by the spaCy run (milliseconds), which this PR doesn't change.

Known limitation (possible follow-up)

A single-sentence Hindi/Bengali paragraph can still become a Title, e.g. "दिल्ली में सरकार की बैठकें होती हैं।", "भारत एक विशाल देश है।" or "রবীন্দ্রনাথ ঠাকুর নোবেল পুরস্কার পেয়েছিলেন।". Both character checks now pass, but with fewer than 2 sentences is_possible_narrative_text runs the English-only verb check (contains_verb), whose result for Hindi depends on spaCy happening to tag some word as an English verb.

partition_text detects hin (it's in element.metadata.languages) but doesn't pass languages to is_possible_narrative_text; is_possible_narrative_text(t, languages=["hin"]) returns True. I'm happy to send that as a separate PR.

Overlap with #4679

#4679 reorders the under_non_alpha_ratio and exceeds_cap_ratio calls inside is_possible_narrative_text. This PR only changes the bodies of those two functions (plus new helpers above is_possible_narrative_text), so the two shouldn't conflict, and the reorder doesn't change any result here since both checks can only make the function return False early.