feat(reindex): verify ported documents before the index swap

Merged
SU
1mo ago
+1135-27

Description

  • The reindex swap now confirms the new index actually contains documents the port reported copying, instead of trusting the port's own success status. A connector or user whose copy never landed no longer gets promoted silently.
  • Covers both port scopes — connector cc_pairs and user files. Previously a user file missing from the new index would not have been noticed.
  • The swap also waits while any index attempt is still IN_PROGRESS against the new settings, and checks again after the sample, so a writer mid-batch cannot race the check. Queued NOT_STARTED attempts do not hold the swap; they run after it against the same settings. A large connector still in its initial indexing therefore delays the swap until that run finishes. A wedged attempt cannot hold it forever: the indexing watchdog fails any attempt with no heartbeat for 30 minutes.
  • A failed check holds the swap and waits PORT_SWAP_VERIFY_RETRY_DELAY_S (default 300 s) before re-checking, so a broken or unreachable index does not make the 15-second swap gate repeat the work indefinitely.
  • Set PORT_SWAP_VERIFY_DOCS_PER_UNIT=0 to skip the document sample. The in-progress-attempt hold still applies, and every other swap condition is unchanged.
  • This is a spot check, not a full comparison. It catches a whole unit missing, a wrong index name, or an emptied index; it will not detect a handful of individually missing documents.

How Has This Been Tested?

External-dependency tests against real Postgres, Redis and OpenSearch (31 in test_port_swap_criterion.py), plus unit tests for the index lookup. The reindex integration tests exercise this path live but need a full deployment, so they are left to CI.

Also run end to end locally: four cc_pairs ported (766 of 766 chunks), the pre-swap sample passed, the swap completed, and the queued index attempt for the new settings started afterwards against the promoted index.

Additional Options

  • [Optional] Please cherry-pick this PR to the latest release version.
  • [Optional] Override Linear Check