fix(csv): read CSV and TSV cells as the file holds them
Summary
read_delimited_text(), which reads every CSV and TSV upload, calls pd.read_csv() with Pandas' default cell interpretation. That changes what the file says in three ways:
| cell | main | this PR |
|---|---|---|
N/A, NULL, None (and NA, nan, n/a, …) | empty | N/A, NULL, None |
12 in a column that has an empty cell, with include_header=True | 12.0 | 12 |
007 in a column of codes that has an empty cell, with include_header=True | 7.0 | 007 |
Probe file name,qty,status,code / Pen,12,N/A,007 / Ink,,NULL,010 / Pad,7,ok,. On main with include_header=True it reads name qty status code Pen 12.0 7.0 Ink 10.0 Pad 7.0 ok, and with include_header=False it reads name qty status code Pen 12 007 Ink 010 Pad 7 ok. With this PR it reads name qty status code Pen 12 N/A 007 Ink NULL 010 Pad 7 ok. Without a header row, the header strings already make each column object, so only the N/A / NULL loss shows there.
Change
Both pd.read_csv() calls in read_delimited_text() pass dtype=str, keep_default_na=False. Every cell is the text the file holds, and an empty field is an empty string. The HTML table still writes an empty cell as empty. Delimiter sniffing, the lone-\r reader and the cell-count limit are unchanged.
Related: #4505 does the same for XLSX (keep_default_na=False), and #4556 keeps XLSX whole numbers (dtype=object). Each PR only adds read options to its own reader.
Testing
- New
test_partition_csv_reads_each_cell_as_written(bothinclude_headervalues) andtest_partition_tsv_reads_each_cell_as_writtenfail onmain. - The rest of
test_csv.pyandtest_tsv.pypasses (97 in total). - The 12 CSV/TSV files in
example-docspartition to exactly the same element text and HTML as onmain. - ruff 0.15.10 check and format are clean, and
scripts/version-sync.sh -cpasses with0.27.34-dev1.