fix(csv): read CSV and TSV cells as the file holds them

Merged
L4
12d ago
+29-4

Summary

read_delimited_text(), which reads every CSV and TSV upload, calls pd.read_csv() with Pandas' default cell interpretation. That changes what the file says in three ways:

cellmainthis PR
N/A, NULL, None (and NA, nan, n/a, …)emptyN/A, NULL, None
12 in a column that has an empty cell, with include_header=True12.012
007 in a column of codes that has an empty cell, with include_header=True7.0007

Probe file name,qty,status,code / Pen,12,N/A,007 / Ink,,NULL,010 / Pad,7,ok,. On main with include_header=True it reads name qty status code Pen 12.0 7.0 Ink 10.0 Pad 7.0 ok, and with include_header=False it reads name qty status code Pen 12 007 Ink 010 Pad 7 ok. With this PR it reads name qty status code Pen 12 N/A 007 Ink NULL 010 Pad 7 ok. Without a header row, the header strings already make each column object, so only the N/A / NULL loss shows there.

Change

Both pd.read_csv() calls in read_delimited_text() pass dtype=str, keep_default_na=False. Every cell is the text the file holds, and an empty field is an empty string. The HTML table still writes an empty cell as empty. Delimiter sniffing, the lone-\r reader and the cell-count limit are unchanged.

Related: #4505 does the same for XLSX (keep_default_na=False), and #4556 keeps XLSX whole numbers (dtype=object). Each PR only adds read options to its own reader.

Testing

  • New test_partition_csv_reads_each_cell_as_written (both include_header values) and test_partition_tsv_reads_each_cell_as_written fail on main.
  • The rest of test_csv.py and test_tsv.py passes (97 in total).
  • The 12 CSV/TSV files in example-docs partition to exactly the same element text and HTML as on main.
  • ruff 0.15.10 check and format are clean, and scripts/version-sync.sh -c passes with 0.27.34-dev1.