fix(docx): classify paragraphs using document languages

CP
20h ago
+52-3

Fixes #3044.

partition_docx now passes the document languages to is_possible_narrative_text when it classifies a paragraph. Before, the call used the default languages=["eng"], so the English sentence-count and verb checks ran on every paragraph. A Danish sentence such as "Dette er et eksempel på en kort sætning." became Text instead of NarrativeText, even with languages=["dan"], because that argument only set metadata.languages.

Changes in unstructured/partition/docx.py:

  • partition_docx declares languages explicitly and passes it to DocxPartitionerOptions. Before, it accepted languages through **kwargs for metadata only.
  • DocxPartitionerOptions.languages returns the caller's languages. When the caller gives none, it detects them from the document text with detect_languages, the same function that sets metadata.languages. It falls back to ["eng"] when detection returns nothing, for example for an empty document or languages=[""].
  • The paragraph classifier passes these languages to is_possible_narrative_text.

partition_doc and partition_odt forward their kwargs to partition_docx, so they get the same behaviour.

Design choices:

  • The fix is in the docx partitioner, not text_type.py. The issue suggests that is_possible_narrative_text skip the English verb check unless language_checks is set. All partitioners call it with language_checks=False, so that would also turn off the verb check for English text everywhere.
  • Languages are detected when the caller gives none. This fixes the default partition_docx(filename=...) and partition(...) calls, and classification agrees with metadata.languages. If you prefer explicit languages only, DocxPartitionerOptions.languages can return self._languages or ["eng"] instead.
  • language_checks stays off. Turning it on would add an English-vocabulary requirement for English documents, which is a separate change.

The text, pptx, xlsx and html partitioners also call is_possible_narrative_text without languages. They are out of scope here.

Two new tests in test_unstructured/partition/test_docx.py partition a Danish paragraph, one with languages=["dan"] and one with no languages. Both fail on main and pass with this change.