fix(email): keep non-ASCII sender and recipient names readable

Merged
US
15d ago
+66-5

What

partition_email() now keeps non-ASCII sender and recipient names, and non-ASCII addresses, as written in the sent_from, sent_to, cc_recipient and bcc_recipient metadata. Closes #4542.

From: José García <jose@ejemplo.es>      (RFC 2047-encoded on the wire, as mail clients write it)
To:   张伟 <zhang@example.cn>

Before: sent_from=['=?utf-8?b?Sm9zw6kgR2FyY8OtYQ==?= <jose@ejemplo.es>']  sent_to=['=?utf-8?b?5byg5Lyf?= <zhang@example.cn>']
After:  sent_from=['José García <jose@ejemplo.es>']                     sent_to=['张伟 <zhang@example.cn>']

From: José <josé@ejemplo.es>               (RFC 6532 address)
Before: UnicodeEncodeError, no elements
After:  sent_from=['José <josé@ejemplo.es>']

Why

The headers are already decoded (email.policy.default), but the four address properties formatted them with email.utils.formataddr(). It writes a non-ASCII display name as an RFC 2047 encoded-word, and it raises UnicodeEncodeError on a non-ASCII address. They now use _format_address(), which quotes and escapes the display name with the same rules as formataddr() but leaves non-ASCII text as is. For ASCII names and addresses the result is unchanged: on 50,000 random ASCII name/address pairs it matched formataddr() exactly.

Validation

  • test_partition_email_keeps_non_ascii_display_names_readable covers From, To (Chinese and German names), a quoted Cc name with a comma, and Bcc. test_partition_email_accepts_non_ascii_addresses covers an RFC 6532 From and To. Both fail on main (the encoded-word, and the UnicodeEncodeError) and pass here.
  • test_unstructured/partition/test_email.py: 75 passed, 1 skipped. test_msg.py gives the same result as on main: one test fails locally on both because the image and PDF extras aren't installed.
  • ruff check and ruff format --check pass on the changed files.
  • CHANGELOG entry under 0.27.27, with the version bumped.