fix(csv): read a field a row does not have as an empty cell, not "None"

SI
20d ago
+24-2

Summary

A CSV row that stops short of the last column has no field there, but partition_csv() wrote the literal word "None" into that position — in element.text and in metadata.text_as_html:

partition_csv(file=io.BytesIO(b"a,b,c\n1,2,3\n4,5\n"))[0]
# before: text 'a b c 1 2 3 4 5 None'   html '<tr><td>4</td><td>5</td><td>None</td></tr>'
# after:  text 'a b c 1 2 3 4 5'        html '<tr><td>4</td><td>5</td><td/></tr>'

That is content the file does not contain, and it reaches chunking, embeddings and prompts.

The cause is an engine difference: ragged rows make the whole-file csv.Sniffer fail to settle on a delimiter, so read_delimited_text() sniffs the delimiter from the first line and lets Pandas read with the Python engine — which hands that missing field back as None, even under dtype=str. DataFrame.to_html() writes a None cell as the hard-coded text "None" before na_rep or a formatter is consulted, and dataframe_to_html() renders whatever the frame holds. The identical bytes read as TSV (delimiter always known → C engine) give "", so CSV and TSV disagreed on the same data.

The fix takes the frame from whichever engine path ran and fills missing cells with the empty string, so a field the row does not have reads as the empty text the file holds. This is not a regression — the same repro reproduces with csv.py from before #4574.

Validation

  • New test test_partition_csv_reads_a_field_a_row_does_not_have_as_an_empty_cell in test_unstructured/partition/test_csv.py asserts both text and text_as_html. It fails on the default branch (assert 'a b c 1 2 3 4 5 None' == 'a b c 1 2 3 4 5') and passes with the fix.
  • pytest test_unstructured/partition/test_csv.py test_unstructured/partition/test_tsv.py → 111 passed, with 4 pre-existing environment errors on Windows (long environment variable), identical with the fix reverted.
  • Lint with the uv.lock-pinned ruff 0.15.10: ruff check and ruff format --check clean on both changed files.

Checklist

  • The PR title is a user-facing release-note entry with a conventional commit prefix.
  • Apply a category label (bug) — applied with this PR.
  • No breaking change.