improvement(knowledge): walk connector reconciliation by id so seen stamps stay off the index
Merged
WA
1mo ago+29959+29959+01234567890123456789012345678901234567890123456789-167-167-012345678901234567890123456789
Summary
- Connector sync stamps
document.source_seen_aton every listed document.source_seen_atis a key column ofdoc_connector_reconciliation_idx, so every stamp was a non-HOT update that wrote every index ondocument. This is release A of two: move every reader off that key so release B can drop it and the stamps become HOT - New
doc_connector_reconciliation_v2_idx (connector_id, id)with the same partial predicate, built concurrently. The old index stays until release B - The absence walks (ACL revoke, soft delete, hard delete) and the member resurrection walk page by id instead of
(seen, id), with absence as a plain filter. Each window is built by a recursive keyset walk that fetches one row per step (id > previous ORDER BY id LIMIT 1) up to 5,000 ids, so every statement reads at most one window whatever plan the database picks. A singleORDER BY id LIMITcould be planned as a bitmap read of the whole connector plus a sort. Matches are filtered within the window, so cost is bounded by ids scanned, not matches found. A walk whose absence count is zero is skipped - Both seen stamps skip rows already stamped at or after the run start (
staleSeen), so current rows aren't rewritten and a later stamp is never overwritten by an earlier one
Type of Change
- Improvement
Testing
- New listing-continuation case: 15k present rows, then a few absent live rows and tombstones last in id order. Every absent row is processed, and every page statement (re-run under EXPLAIN ANALYZE) stays within the window bound. It fails on the unbounded id keyset (15,003 rows scanned in one statement)
- Seen-stamp guard: rows stamped after the run start keep their stamp (connector-lease-pages); it fails with the guard removed
- Scale suite: both window statements use the v2 index. Reconciliation took 2.0s vs 15.9s with unbounded pages
- listing-continuation (incl. NULL and microsecond tie cases), member-document-lifecycle, sync-content-pass, and connector-lease-pages pass, as do the
lib/knowledgeunit tests, type-check, lint,check:audits, andcheck:migrations origin/staging - Release B, after this deploys: drop
doc_connector_reconciliation_idxconcurrently and add a HOT-update regression test - Migration renumbered to 0388 on top of staging
Checklist
- Code follows project style guidelines
- Self-reviewed my changes
- Tests added/updated and passing
- No new warnings introduced
- I confirm that I have read and agree to the terms outlined in the Contributor License Agreement (CLA)