fix: stop dropping nested metadata values in convert_to_csv

LI
24d ago
+238-4

Summary

Fixes #4548.

convert_to_csv() exported the data_source and enrichment_origins columns (they are part of TABLE_FIELDNAMES via ElementMetadata.__annotations__), but their values were always empty.

Root cause

Nested metadata (e.g. data_source, enrichment_origins) is flattened by flatten_dict() into leaf sub-keys such as data_source_url, data_source_version, enrichment_origins_ner. The CSV writer then filtered those flattened keys against table_fieldnames, which only contains the raw dict keys (data_source, enrichment_origins). Because the sub-keys never matched the whitelist, they were dropped and the columns stayed empty — a silent data-loss bug in production.

This aligns CSV export with convert_to_dataframe(), which already merges the flattened metadata keys directly.

Change

  • Collect flattened metadata sub-keys across rows and append them to the fieldnames in order, so they land as CSV columns with the real values.
  • Backward compatible: the fixed TABLE_FIELDNAMES column set is unchanged; only previously-dropped sub-keys are added.

RED / GREEN

  • Before the fix: test_convert_to_csv_includes_flattened_nested_metadata fails with KeyError: 'data_source_url'.
  • After the fix: the regression test passes; the full test_unstructured/staging/test_base.py suite (86 tests) passes.

Testing

from unstructured.documents.elements import DataSourceMetadata, ElementMetadata, NarrativeText
from unstructured.staging.base import convert_to_csv

el = NarrativeText(
    "Hello",
    metadata=ElementMetadata(
        data_source=DataSourceMetadata(url="s3://bucket/report.pdf", version="v3"),
        enrichment_origins={"ner": [{"mode": "Auto"}]},
    ),
)
print(convert_to_csv([el]))

Expected: data_source_url, data_source_version, enrichment_origins_ner columns are present and populated.

AI assistance

This contribution was prepared with AI assistance; no project AI policy was found, so disclosing here.