feat(indexing): chunk spreadsheets strategy w/ less chunks

Merged
DA
6d ago
+2709-1868

Problem

Spreadsheets were chunked row by row, so chunk count grew with row count. The 30-workbook benchmark corpus produced 50,976 chunks. Nobody retrieves on a row of digits, and the volume costs embedding time and index space.

The strategy

A workbook is chunked once, as a whole, rather than a sheet or a row at a time. DocumentChunker groups a document's TabularSections by link and hands each group to TabularChunker.chunk_workbook, so the chunks can describe one file and stay bounded in number. Two attachments on one document stay two files.

workbook_chunks.py:build_workbook_chunks builds five kinds of block:

blockscopechunks
overview — names every sheet, its size and its columnsone per workbookup to 3
profile — what a large sheet contains and totals toone per large sheetup to 2
verbatim rows — every row, unchangedone per small sheetup to 3
text sheet — a Readme too long for the overview's About:one per text sheetup to 3
more values — frequent values of free-text columnsper large sheet1 each

A sheet counts as small when all its rows fit the verbatim budget (TABULAR_FULL_ROWS_MAX_CHUNKS × the token limit); otherwise it gets a profile. Blocks are queued one per sheet before any sheet gets extras, then packed greedily so small blocks share a chunk. TABULAR_MAX_CHUNKS_PER_WORKBOOK (default 24) is the ceiling, and whatever it drops is logged.

Result on the same corpus: 105 chunks, ~485x fewer, at most 7 per workbook, none empty and none over the token budget.

What the chunks look like

Overview — one per workbook, the index of the file:

Spreadsheet {file}: {N} sheets ({name}, {name}, …)
About: {text-sheet lines and title blocks, ≤150 tokens}            ← optional
Sheet {name}: {R} rows; Columns: {col} ({alias}), …                ← one per table sheet

Profile — one per large sheet. Every figure is exact over all rows; what is partial is the row-level detail:

{file} :: {sheet}: {R} rows x {C} columns (summary; not every row is shown)
Columns: {col} ({alias}), …
Time range: {col} {min} to {max}; …                                ← date columns
Identifiers: {col} {min} … {max} ({n} distinct); …                 ← identifier columns
{col} values (count): {v} ({n}); …                                 ← one per category column
Totals across all rows:
{col}: total {t}{unit}; average {a}{unit}; min {m}{unit}; max {M}{unit}; count {n}
Total {numeric} by {category} (highest first): {v} {sum} (avg {avg}); …
Total {numeric} by month: {YYYY-MM} {sum}; …
{col} ({n} distinct; most common first): {v}; …                    ← free-text columns
Sample row: {col}={v}, …                                           ← up to 2

A % column says Average … by … instead, since percentages do not add up. A clipped line ends with (43 of 120 months shown).

Verbatim rows — one per small sheet. The two heading lines repeat at the top of each chunk:

{file} :: {sheet} (all {R} rows)
Columns: {col} ({alias}), …
{col}={v}, {col}={v}, …                                            ← every row

Text sheet and more values:

{file} :: {sheet}                        {file} :: {sheet}: more values
{cell} | {cell} | …                      {col}: {v}; {v}; …

A real workbook, end to end

frtr_0002_ops-inventory.xlsx — 5 sheets, 9,720 rows → 7 chunks: an overview, three of verbatim Vendor rows, two profiles, and the KPI rows.

Spreadsheet frtr_0002_ops-inventory.xlsx: 5 sheets (Readme, Vendors, PurchaseOrders, InventoryMovements, KPIs)
About: FRTR – Operations & Inventory (Medium) Mid-size operations workbook with inventory movements, purchase orders, vendor mapping, KPIs, and embedded images.
Sheet Vendors: 120 rows; Columns: VendorID (Vendor ID), VendorName (Vendor Name)
Sheet PurchaseOrders: 3500 rows; Columns: Date, POID, SKU, VendorID (Vendor ID), Qty, UnitCost (Unit Cost), TotalCost (Total Cost)
Sheet InventoryMovements: 6000 rows; Columns: Date, SKU, Type, Qty, UnitCost (Unit Cost), Value
Sheet KPIs: 50 rows; Columns: SKU, VendorID (Vendor ID), VendorName (XLOOKUP), …

frtr_0002_ops-inventory.xlsx :: InventoryMovements: 6000 rows x 6 columns (summary; not every row is shown)
Columns: Date, SKU, Type, Qty, UnitCost (Unit Cost), Value
Time range: Date 2024-01-01 to 2024-12-31
Identifiers: SKU SKU-00100 ... SKU-00399 (300 distinct)
Type values (count): OUT (3020); IN (2980)
Totals across all rows:
Qty: total -4263; average -0.7105; min -200; max 200; count 6000
Value: total -123074.90; average -20.51; min -8839.58; max 8794.08; count 6000
Total Qty by Type (highest first): IN 299430 (avg 100.5); OUT -303693 (avg -100.6)
Total Qty by month: 2024-01 -681; 2024-02 -110; 2024-03 -3282; 2024-04 4192; …
Sample row: Date=2024-06-17, SKU=SKU-00207, Type=OUT, Qty=-5, UnitCost=36.9, Value=-184.5

The trade

Rows reach the index only through a small sheet's verbatim block or a profile's sample rows. A query naming one row of a large sheet has little to match. The chunks make a file findable and say what it holds; the file itself stays the record.

This replaces the row path rather than sitting beside it, so behaviour changes on deploy with no flag, and existing indexes keep their old row chunks until re-indexed.

Correctness decisions worth review

  • Dates carry no locale assumption. 5/3/2024 is 5 March to most of the world and May 3 in the US, and the separator settles nothing. The order is inferred per column from its own values — a part above 12 cannot be a month — with both readings tracked until one is ruled out. A column that never settles gets no time range and no monthly totals, and logs why.
  • Nothing is summed that does not add up. A % column reports average, min and max; a column mixing currencies, or two tabs measuring one column in different ones, are never pooled into a total. Every figure carries its unit.
  • Clipped lines say so, e.g. (43 of 120 months shown), so a truncated breakdown is not read as the whole truth.
  • A title block no longer hides the table. Narrow preamble rows are skipped to reach the header and kept for the overview. Previously a table behind 25 metadata rows was classified as a text sheet and lost 82% of its data.
  • Headers are read by cell shape. Two or more year columns above a row with none is a heading (Region,2023,2024); a lone number between 1900 and 2199 is an amount, and PAY-9 above PAY-10 is the next record.
  • The chunk budget costs detail, not whole sheets. One block per sheet queues before any sheet gets extras, the overview names every sheet, and the packer logs whatever it drops.

Shape of the change

workbook_chunks.py (879 lines) builds the blocks, cell_values.py (362) reads one cell, models.py (414) holds the data shapes. TabularChunker drops from 375 lines to 129.

Removed: the row path and its descriptors — analysis.py, sheet_descriptor.py, total_descriptor.py, util.py and the 1,134-line test_tabular_section_chunker.py.

csv_utils gains csv_reader, which raises the 128 MiB field size limit at the point of read rather than on import. The old module-level call meant correctness depended on import order, and tabular_section_utils had been converting TSV at the 128 KiB default all along.

Testing

83 tests in test_workbook_chunks.py; 270 pass across indexing, csv_utils and configs. The full unit suite is green apart from pre-existing test_process_isolation.py failures. Measured end to end over 30 real workbooks: counts above, no empty output, no chunk over budget.

Known follow-ups

  • No retrieval-quality evidence yet. The chunk reduction is measured; answer quality is not. The existing harness scores a filename match at document level, so it cannot detect right-sheet-wrong-row, which is this change's main risk.
  • Wide sheets lose content silently. On a 300-column sheet the Columns: line truncates mid-word and only a fraction of the totals lines survive the profile's 2-chunk cap, with no marker.
  • The old descriptor's aggregate vocabulary is gone ("total, overall, grand total, sum across all, average, …"). Earlier benchmarks had the descriptor variant winning, so restoring that phrasing is worth testing.
  • The "more values" block never fires on any corpus workbook, and carries no context or truncation marker when it does.
  • Verified only against openpyxl extraction; a real indexing run should confirm what Onyx's own xlsx path feeds the chunker, particularly for formula cells.

🤖 Generated with Claude Code

Summary by cubic

Chunks each spreadsheet once as a bounded per-workbook profile instead of row by row. On the 30-workbook benchmark corpus the chunk count drops from 50,976 to 107 (476x fewer), with no file over TABULAR_MAX_CHUNKS_PER_WORKBOOK (default 24). This replaces the row path rather than sitting beside it, so behavior changes on deploy with no flag, and existing indexes keep their old row chunks until re-indexed.

Chunk contents

  • Each workbook is chunked once at its first TabularSection: an overview naming every sheet, verbatim rows for small sheets, and a profile per large sheet (column roles, identifier ranges, totals, breakdowns by category and by month, frequent values, sample rows).
  • Large sheets with the same columns share one profile, and sheets are grouped by file link so two attachments in one document are never merged into a workbook that doesn't exist.
  • Rows reach the index only through a small sheet's verbatim block or a profile's sample rows, so a query naming one row of a large sheet has little to match.
  • Date order is inferred per column from its own values with no locale assumption; percentage columns report average, min and max, never a sum; truncated breakdowns say so.
  • Header rows are detected by cell shape: two or more years above a row with no numbers read as headings, while a lone year between 1900 and 2199 stays a value.

Removals, config, and known gaps

  • Removes the row path and its descriptor modules (analysis.py, sheet_descriptor.py, total_descriptor.py, util.py) and its 1,134-line test file, replaced by test_workbook_chunks.py; TabularChunker drops from 375 to 129 lines.
  • Chunk building lives in workbook_chunks.py, with cell-level parsing in cell_values.py and the data models in models.py.
  • csv_utils gains csv_reader, which sets the 128 MiB field size limit at the point of read instead of on import, so long cells no longer depend on import order; all CSV reads in the backend now go through it.
  • Three new env vars cap full-row chunks per sheet, frequent-value chunks per sheet, and total chunks per workbook; they replace the unreleased TABULAR_SKETCH_* names.
  • Column parsing drops its hardcoded abbreviation, unit-suffix, currency-symbol, and month-name tables: aliases split on words, every Unicode currency symbol is recognized, and any 3+ letter month prefix resolves.
  • Aggregates are exact over every row; grouped percentage figures report averages and every grouped figure carries its unit, mixed-currency columns are never summed, and tabs in different currencies are not pooled.
  • Known gaps: no retrieval-quality evidence for the reduction yet, wide sheets lose columns and totals without a marker, and the old descriptor's aggregate vocabulary is gone.

Summary by CodeRabbit

  • New Features

    • Spreadsheet and CSV content is organized across each workbook, with a limit on the number of chunks per file.
    • Small tables can retain their rows, while larger tables provide profiles, sample data, summaries, and frequent values.
    • Workbook overviews summarize sheets, and compatible sheets with matching headers may be combined.
    • Profiles include totals, date ranges, and category or monthly breakdowns when supported by the data.
    • Ambiguous dates are handled conservatively, and details may be clipped to fit content limits.
    • Sheets are processed without loading them entirely into memory. Files with no extracted chunks remain findable by title and metadata.

  • Bug Fixes

    • CSV readers can handle larger individual fields.