fix(csv): read a repeated header label as written
Summary
read_delimited_text(), which reads every CSV and TSV upload, gets its header labels from pd.read_csv(), which makes a repeated label unique as it reads it. A header row name,name,age therefore yields the labels name, name.1, age; when a genuine name.1 label is also present, Pandas names the repeated name column name.2. The fabricated labels appear in the Table element's text and metadata.text_as_html, the same class of corruption #4574 fixed for data cells.
| header row | main labels | this PR labels |
|---|---|---|
name,name,age | name name.1 age | name name age |
name,name,name.1 | name name.2 name.1 | name name name.1 |
Fixes #4708.
Change
read_delimited_text() peeks the first header line once (the delimiter sniff already reads it) and, after the read, _restore_repeated_header_labels() gives the data-frame back the labels the file holds when they repeat. An empty header cell keeps the Unnamed: n label Pandas gives it, which is the current emitted behavior (pinned by existing tests). No pandas option can disable the renaming (mangle_dupe_cols was removed in pandas 2.0), and reassembling the read with header=None would change the index and Unnamed: n semantics, so the labels are restored after the read.
The restoration is best-effort and gives up (leaving Pandas' labels) rather than mis-read or fail, per the review:
- the header line is peeked within a bounded budget of 1 Mi decoded characters inspected, including a blank prefix and text already queued by delimiter detection. An oversized header or prefix leaves Pandas' labels in place before the new splitter allocates fields; the existing new-fill budget and unbounded delimiter-sniffing behavior are unchanged;
- the peek skips only the lines the read itself skips, so a header record of empty tab-separated cells (
\t\nA\tA\n) keeps itsUnnamedlabels instead of taking the data row's values — on the sniffed path the line is re-peeked with the sniffed delimiter; - a header line that does not hold a complete record, a quoted cell spanning lines, is left alone (its field count alone cannot tell, when the spanning cell is last);
- the record is split by
_split_header_record(), an in-module quote-aware scanner, instead ofcsv.reader, so a field past the csv module's field-size limit is restored too and the labels no longer depend on the process-wide limitpartition_csv()raises for its own read — no global CSV state is read or set on this path (per the re-review).
Compatibility
Restoring the file's repeated labels changes Table.text and metadata.text_as_html for any file whose header row repeats a label — that is the fix. For documents re-processed after this change, the changed text also changes any default element ID derived from that text, so a consumer that upserts by ID may need to reconcile or re-ingest the older elements (raised in review). Files whose header labels do not repeat keep main's output, pinned by the existing tests. Whether to accept this behavior change, and what release or migration guidance to attach, is for the maintainers to decide.
Validation
test_unstructured/partition/test_csv.py and test_tsv.py: 136 passed, 9 skipped, covering a repeated label (text and text_as_html), a genuine name.1 label, a repeated label beside an empty header cell (Unnamed: n kept), a quoted repeated label, CRLF endings, and — from the review rounds — a quoted header cell spanning lines (labels left alone, no text lost), a header record of empty cells (explicit and sniffed delimiter), a blank prefix past the peek budget, the budget itself (white-box), label restoration independent of the ambient field-size limit (same text and HTML at a 1 KiB limit and at the ambient one), and a differential test checking the in-module splitter against the csv module over every single-line record it can be reached with (strings through length 7 over a four-symbol alphabet, three line endings, both delimiters). The first seven fail on main with the fabricated label; the first review round's tests fail on this PR's first commit and pass on the second; the re-review's tests fail on the second and pass on the third. The latest review adds 14 parameterized cases for a prefilled header/prefix (including an unterminated line), the budget boundary, input replay after giving up, and the sniffed partition_csv() path with large non-ASCII repeated/unique labels; those same cases produce 9 failures and 5 passes on 56c11c0 and all pass with the fix. The fourth review round's regression holds only while the unbounded sniff sample is still bound through pd.read_csv() on a headerless sniffed read (red on 5b54e9b, green with the sample back to a temporary); nothing else changes. Repository-wide ruff check . and ruff format --check . pass. The branch includes current main (ad56303); its separate detection-metrics regression test could not collect in this CSV test environment because torch is not installed.