improvement(knowledge): walk connector reconciliation by id so seen stamps stay off the index

Merged
WA
1mo ago
+29959-167

Summary

  • Connector sync stamps document.source_seen_at on every listed document. source_seen_at is a key column of doc_connector_reconciliation_idx, so every stamp was a non-HOT update that wrote every index on document. This is release A of two: move every reader off that key so release B can drop it and the stamps become HOT
  • New doc_connector_reconciliation_v2_idx (connector_id, id) with the same partial predicate, built concurrently. The old index stays until release B
  • The absence walks (ACL revoke, soft delete, hard delete) and the member resurrection walk page by id instead of (seen, id), with absence as a plain filter. Each window is built by a recursive keyset walk that fetches one row per step (id > previous ORDER BY id LIMIT 1) up to 5,000 ids, so every statement reads at most one window whatever plan the database picks. A single ORDER BY id LIMIT could be planned as a bitmap read of the whole connector plus a sort. Matches are filtered within the window, so cost is bounded by ids scanned, not matches found. A walk whose absence count is zero is skipped
  • Both seen stamps skip rows already stamped at or after the run start (staleSeen), so current rows aren't rewritten and a later stamp is never overwritten by an earlier one

Type of Change

  • Improvement

Testing

  • New listing-continuation case: 15k present rows, then a few absent live rows and tombstones last in id order. Every absent row is processed, and every page statement (re-run under EXPLAIN ANALYZE) stays within the window bound. It fails on the unbounded id keyset (15,003 rows scanned in one statement)
  • Seen-stamp guard: rows stamped after the run start keep their stamp (connector-lease-pages); it fails with the guard removed
  • Scale suite: both window statements use the v2 index. Reconciliation took 2.0s vs 15.9s with unbounded pages
  • listing-continuation (incl. NULL and microsecond tie cases), member-document-lifecycle, sync-content-pass, and connector-lease-pages pass, as do the lib/knowledge unit tests, type-check, lint, check:audits, and check:migrations origin/staging
  • Release B, after this deploys: drop doc_connector_reconciliation_idx concurrently and add a HOT-update regression test
  • Migration renumbered to 0388 on top of staging

Checklist

  • Code follows project style guidelines
  • Self-reviewed my changes
  • Tests added/updated and passing
  • No new warnings introduced
  • I confirm that I have read and agree to the terms outlined in the Contributor License Agreement (CLA)