fix(lang): read region-qualified language tags like en-US

AB
10h ago
+20-1

Summary

detect_languages() cuts each language the caller passes down to its first three characters before the ISO 639 lookup. That works for eng, en or english, but a region-qualified tag such as en-US, pt-BR or zh-CN becomes en-, pt- or zh-, matches nothing, and is dropped with a warning. Elements then end up with metadata.languages == [] even though the caller said which language the document is in.

detect_languages("This text is long enough to skip the short text rule.", ["en-US"])
# before: []
# after:  ["eng"]

The fix looks up the part before the hyphen (en-US -> en), then keeps the existing three-character slice, so names like english and Tesseract codes like chi_sim behave as before.

Validation

  • Added test_detect_languages_reads_region_qualified_language_tags (en-US, pt-BR, zh-CN, en-GB + fr-CA). It fails on main and passes with the fix.
  • pytest test_unstructured/partition/common/test_lang.py: 69 passed, 4 skipped.
  • ruff check and ruff format --check pass on the changed files.

Checklist

  • The PR title is a user-facing release-note entry with a conventional commit prefix.
  • Apply a category label (enhancement, bug, or an appropriate existing label), or ask a maintainer to apply it.
  • For breaking changes, use ! in the title and explain impact and migration above.