feat(indexing): re-chunk tabular documents indexed by the old chunker (#16153) to release v4.8

Merged
DA
3d ago
+1138-14

Cherry-pick of #16153 to release v4.8 — the migration that re-chunks tabular documents already indexed by the old chunker.

Upstream commit: e6772d6f417e4289b3e5cd9f0c9c19abdd0a7ece

Conflict resolution — mostly neighbouring main-only entries this branch does not have:

  • app_base.py, constants.py, beat_schedule.py: kept this branch's entries and added only the tabular_rechunk ones. Upstream's cc_pair_ids_backfill task, KV key and lock are not on this branch and were not carried over.
  • env templates and values.yaml: took the TABULAR_RECHUNK_ENABLED entries, dropped the SKIP_SPREADSHEET_CONTENT ones (not a setting on this branch).
  • Chart.yaml: bumped this branch's version 0.8.28 → 0.8.29 and replaced the change list, per the convention on this branch.
  • db/document.py: kept both sides of the import block (KGEntity/KGRelationship plus the new FileRecord), and added true to the SQLAlchemy imports — get_oversized_tabular_documents joins with it, and nothing on this branch imported it yet. The cherry-pick also pulled in PortedScope and sample_ported_document_ids as context; those belong to #14463, which is not on this branch, so they were removed.

Adapted to this branch's document index, which still has both Vespa and OpenSearch:

  • run_tabular_rechunk.py imports DocumentSectionRequest from interfaces_new (upstream's interfaces is the post-#15336 name).
  • It resolves the index with get_default_document_index(...) instead of build_opensearch_document_index(...), typed DocumentIndex. Hardcoding OpenSearch would read chunk heads from an index a Vespa-serving deployment never populates, and every document would be skipped as "not re-chunkable". id_based_retrieval has the same signature on both backends. test_run_tabular_rechunk.py patches the new name.
  • tabular_rechunk/tasks.py: dropped the SKIP_SPREADSHEET_CONTENT gate from _blocked_reason, since that setting does not exist here. The other gates (disabled vector DB, reindex in flight, port backfilling) are unchanged.

Verified: tests/unit/onyx/background and tests/unit/onyx/indexing pass (303 passed, 6 skipped); ruff check clean on every file this touches. test_oversized_tabular_scan.py came across unchanged and was not run here (it needs Postgres).

Stacked backport, merge last. Order: #16296 → #16298 → #16308 → #16309 → this PR.

Summary by cubic

Backports #16153, the one-time migration that re-chunks tabular documents indexed by the old chunker, to v4.8. With TABULAR_RECHUNK_ENABLED set, a beat task re-chunks oversized tabular documents in place from the stored workbook, without re-crawling the source; a KV record stores the cursor and completion state, so it resumes across ticks and never re-runs once finished.

Backport adaptations

  • Left out upstream content not on this branch: the cc_pair_ids_backfill task, KV key, lock, and the PortedScope pieces from #14463.
  • The run imports DocumentSectionRequest from interfaces_new (upstream's interfaces is the post-#15336 name), and resolves the index with get_default_document_index(...) typed DocumentIndex instead of build_opensearch_document_index(...): this branch serves both Vespa and OpenSearch, and hardcoding OpenSearch would skip every document under a Vespa-serving deployment.
  • The SKIP_SPREADSHEET_CONTENT gate was dropped from _blocked_reason because that setting does not exist here; the remaining gates (disabled vector DB, reindex in flight, port backfilling) are unchanged.
  • db/document.py keeps both import blocks and adds true to the SQLAlchemy imports; miscellaneous_utils.py gains experts_from_stores_representations for the owner round-trip.

Verified: tests/unit/onyx/background and tests/unit/onyx/indexing pass (303 passed, 6 skipped), and ruff check is clean on every touched file. test_oversized_tabular_scan.py needs Postgres and was not run.

  • [Optional] Override Linear Check