fix: stop dropping nested metadata values in convert_to_csv
Summary
Fixes #4548.
convert_to_csv() exported the data_source and enrichment_origins columns (they are part of TABLE_FIELDNAMES via ElementMetadata.__annotations__), but their values were always empty.
Root cause
Nested metadata (e.g. data_source, enrichment_origins) is flattened by flatten_dict() into leaf sub-keys such as data_source_url, data_source_version, enrichment_origins_ner. The CSV writer then filtered those flattened keys against table_fieldnames, which only contains the raw dict keys (data_source, enrichment_origins). Because the sub-keys never matched the whitelist, they were dropped and the columns stayed empty — a silent data-loss bug in production.
This aligns CSV export with convert_to_dataframe(), which already merges the flattened metadata keys directly.
Change
- Collect flattened metadata sub-keys across rows and append them to the fieldnames in order, so they land as CSV columns with the real values.
- Backward compatible: the fixed
TABLE_FIELDNAMEScolumn set is unchanged; only previously-dropped sub-keys are added.
RED / GREEN
- Before the fix:
test_convert_to_csv_includes_flattened_nested_metadatafails withKeyError: 'data_source_url'. - After the fix: the regression test passes; the full
test_unstructured/staging/test_base.pysuite (86 tests) passes.
Testing
from unstructured.documents.elements import DataSourceMetadata, ElementMetadata, NarrativeText
from unstructured.staging.base import convert_to_csv
el = NarrativeText(
"Hello",
metadata=ElementMetadata(
data_source=DataSourceMetadata(url="s3://bucket/report.pdf", version="v3"),
enrichment_origins={"ner": [{"mode": "Auto"}]},
),
)
print(convert_to_csv([el]))
Expected: data_source_url, data_source_version, enrichment_origins_ner columns are present and populated.
AI assistance
This contribution was prepared with AI assistance; no project AI policy was found, so disclosing here.