fix(staging): write LabelBox staging text files as UTF-8
Summary
stage_for_label_box() writes each element's text to <external-id>.txt in text mode with no explicit encoding, so the bytes are produced with the platform's locale codec:
https://github.com/Unstructured-IO/unstructured/blob/main/unstructured/staging/label_box.py#L85
with open(output_filepath, "w+") as output_text_file:
output_text_file.write(element.text)
Two consequences:
- It raises where the locale codec cannot represent the text —
cp1252on Windows, or ASCII when the process runs under aPOSIX/Clocale (common in containers). Any non-Latin-1 document (CJK, Greek, an em dash) aborts staging. - Where it does not raise, the staged file is written in the locale encoding while LabelBox reads the URL as UTF-8, so the labelling task silently shows mojibake.
Every other open() in unstructured/staging/ already passes an explicit encoding — the six in unstructured/staging/base.py (elements_from_json, elements_to_json, convert_to_csv/dataframe, elements_to_md, …) all take an encoding argument and forward it. label_box.py is the only one that does not.
Reproduction
On any interpreter whose locale.getpreferredencoding(False) is not UTF-8 (here Windows / cp1252, Python 3.12):
from unstructured.documents.elements import NarrativeText
from unstructured.staging.label_box import stage_for_label_box
stage_for_label_box(
[NarrativeText(text="第一段:表格中的中文内容。")],
output_directory=out,
url_prefix="https://example.com/docs",
create_directory=True,
)
UnicodeEncodeError: 'charmap' codec can't encode characters in position 0-12: character maps to <undefined>
Fix
One line — pass encoding="utf-8", matching the sibling serializers and what LabelBox expects on the other end.
Tests
test_stage_for_label_box_writes_text_files_as_utf8 stages an element containing an em dash, CJK and an accented character, then asserts both that the staged bytes decode as UTF-8 and that the text-mode open() was given an explicit utf-8. The second assertion is what makes the test fail on a UTF-8 platform too, where the old code happens to produce the right bytes.
- before the fix:
1 failed, 13 passed(fails at the write oncp1252; fails on the encoding assertion under a UTF-8 locale) - after the fix:
14 passed
The pre-existing test_stage_for_label_box read the staged file back with a bare open() as well; that read is now explicit too, so the assertion does not depend on the runner's locale.
ruff check / ruff format --check clean on both files, and scripts/version-sync.sh -c passes for the 0.27.3 bump.
Not included
The remaining locale-dependent open() calls in the package live under unstructured/metrics/ (evaluation tooling reading prediction/ground-truth JSON and .txt files). They are the same class of defect but a separate surface — happy to send a follow-up if you want them covered.
🤖 Generated with Claude Code