fix(lang): read region-qualified language tags like en-US
AB
10h ago+20+20+01234567890123456789-1-1-0123456789
Summary
detect_languages() cuts each language the caller passes down to its first three characters before the ISO 639 lookup. That works for eng, en or english, but a region-qualified tag such as en-US, pt-BR or zh-CN becomes en-, pt- or zh-, matches nothing, and is dropped with a warning. Elements then end up with metadata.languages == [] even though the caller said which language the document is in.
detect_languages("This text is long enough to skip the short text rule.", ["en-US"])
# before: []
# after: ["eng"]
The fix looks up the part before the hyphen (en-US -> en), then keeps the existing three-character slice, so names like english and Tesseract codes like chi_sim behave as before.
Validation
- Added
test_detect_languages_reads_region_qualified_language_tags(en-US, pt-BR, zh-CN, en-GB + fr-CA). It fails on main and passes with the fix. pytest test_unstructured/partition/common/test_lang.py: 69 passed, 4 skipped.ruff checkandruff format --checkpass on the changed files.
Checklist
- The PR title is a user-facing release-note entry with a conventional commit prefix.
- Apply a category label (
enhancement,bug, or an appropriate existing label), or ask a maintainer to apply it. - For breaking changes, use
!in the title and explain impact and migration above.