fix(csv): keep every field of CSV and TSV rows wider than the first row

IA
4d ago
+315-47

Summary

Fixes #4682.

partition_csv() and partition_tsv() failed, or silently lost text, when a row had more fields than the first row (or the header):

  • A data row with a trailing delimiter, or a closing "notes" row with extra fields, made the whole file fail with a Pandas ParserError. The repo's own example-docs/single-column.csv does this.
  • With include_header=True, data rows wider than the header had their first fields made the row index by Pandas, and to_html(index=False) dropped them: a,b / 1,2,3 / 4,5,6 came out as a b 2 3 5 6.

Fix. check_cell_count() already scans the whole file to bound the cell count. It now also measures the widest record, with the quoting rules _first_record_width() applies to the first record, and returns that width. read_delimited_text() passes it to Pandas as names=range(width), so every record is read whole and shorter ones are padded with empty cells. With a header, the first row becomes the header and is labeled by Pandas the same way as reading with header=0, so existing Unnamed: 1 and .1 labels don't change. A column the header has no cell over is labeled like an empty header cell (Unnamed: <i>).

Because Pandas is given the width, the table can no longer be wider than what the scan counted: the cell limit now holds by construction, and the implicit-index special case in check_cell_count() is gone. index_col=False is deliberately not passed: with it, the Python engine drops the extra fields of a row wider than names; without it, both engines raise. So a width measured short would fail loudly instead of losing text.

Padding to the widest row means the Python engine fills more cells with None, which to_html() writes as the text "None", so the frame goes through fillna(""). This also changes one existing behavior: a row shorter than the first, read by the Python engine (used when the delimiter is not ,, ; or |), no longer shows "None" in its missing cells. #4596 makes the same change for those short rows, so whichever of the two lands second needs a small rebase.

Validation

  • New tests (fail on main, pass here):
    • exact text and text_as_html for a trailing delimiter, a wide notes row, and data rows wider than the header;
    • the Python-engine case where Pandas made the first data row the index names;
    • a TSV case, with and without a header;
    • a cell-limit test where only the last row is 1,000 fields wide: it is rejected before read_csv() is called, and accepted at exactly 10 × 1,000 cells;
    • a randomized test that compares the measured width, and every cell read, with the records written (quoted delimiters and line breaks, "", a quote inside an unquoted field, text after a closing quote, blank lines, a BOM, \r\n), at chunk sizes 1, 3 and 65,536. I also ran it on 2,800 more random files at chunk sizes 1, 2, 3, 5, 7, 64 and 65,536, with no mismatch.
  • Header labels unchanged: a new test reads headers with both the old header=0 path and the new one, with both engines, and checks that labels and cells match: repeated labels (.1), empty header cells (Unnamed: <i>), a quoted header with a delimiter, "" and a line break, spaces around labels, a label like a.1, and a header-only file. Locally I also compared 31 random well-formed tables the same way, with no difference.
  • Changed test: test_partition_tsv_with_implicit_index_columns_matches_pandas_within_the_limit pinned the output to Pandas' header=0 result, which for that file is h z: Pandas made x and y index names and each a a row index. It now asserts the full text, h Unnamed: 1 Unnamed: 2 x y z a a a, and is renamed to test_partition_tsv_keeps_data_rows_wider_than_the_header_within_the_limit.
  • Unchanged output: for every .csv and .tsv file in example-docs/, with include_header both ways, the output is identical to main, except single-column.csv, which raised ParserError on main and now parses with all its text.
  • Test suites: test_csv.py and test_tsv.py: 206 passed. Together with the file-type, staging, utils and chunking tests, everything else passes on both this branch and main, apart from the same 22 tests in test_filetype.py, which fail on both because my machine has no libmagic. test_auto.py needs the PDF extras, which I don't have installed. make check passes.
  • Speed, on a 900,000-row × 5-column file, best of 3: the cell-count scan goes from 0.05 s to 0.24 s with plain fields, and from 0.07 s to 2.0 s when every field is quoted and contains "". The whole partition_csv() call takes 26 to 30 s on these files.

Checklist

  • The PR title is a user-facing release-note entry with a conventional commit prefix.
  • Apply a category label (enhancement, bug, or an appropriate existing label), or ask a maintainer to apply it. I can't apply labels, so could a maintainer add bug?
  • Not a breaking change.