improvement(knowledge): maintain keyword projections only for search indexes
Merged
WA
1mo ago+645+645+012345678901234567890123456789-56-56-01234567890123456789
Summary
- Since #8314, workspace knowledge search ranks on
embedding/document. The only live reader ofembedding_keyword_searchandembedding_keyword_tinis dormant indexed search, and only for bases that are search indexes. Yet every chunk write still upserted a keyword row, ran a TinDELETEon insert, and every document insert fanned an ACL/connector sync out to the projections - Script migration
0025_scope_keyword_projections, allCREATE OR REPLACEin one transaction with the 0024 lock-timeout retry:sync_embedding_keyword_searchtakes the shared membership lock and writes keyword rows only foris_search_indexbases. On an update that moves a chunk out of a search index, it deletes the chunk's row- A new flip trigger backfills a base's keyword rows when it becomes a search index and deletes them when it stops being one, under the exclusive membership lock. It is separate from the Tin flip trigger, which only exists where Tin is installed. Its upsert skips unchanged rows
- The Tin chunk trigger skips its
DELETEon insert, since a new chunk can't have a Tin row projection_source_acl_syncbecomesAFTER UPDATE OF connector_id, acl ... WHENthe value actually changed. A new document has no chunks yet (FK), so the insert arm never matched anything
- The projector's keyword page writes only for search-index bases, keyed on the base flag
- Fresh installs,
db:push, and late Tin adoption end with the same trigger definitions: 0019 installs the final guarded, key-share-locked Tin triggers, and 0025 re-runs after 0016. Adb:pushre-run of the legacy 0016 backfill may refill keyword rows for non-search bases; they're unread and left in place like the existing ones - Both adoption backfills read their chunks
FOR KEY SHARE, so a chunk delete racing an adoption is waited out instead of failing it - Rollback floor: v0.9.1 and earlier ranked workspace keyword search through
embedding_keyword_search. Once 0025 has run, don't roll back below v0.9.2, and self-hosters should upgrade through v0.9.2 or later - Existing keyword rows for non-search bases are left in place (unread) and removed separately
Type of Change
- Improvement
Testing
- New
0025_scope_keyword_projections.integration.ts: insert scoping, move-out, adopt and un-adopt, commit and uncommitted-adoption races, no fan-out for inserted documents while a real ACL change still fans out, no-op adoption rewrites skipped, and late Tin adoption keeping the guarded trigger. Each fails with its guard reverted - Updated the projector, 0016 backfill, embedding-insert-batches, and processing-lock-scope suites. The search-index fixtures restore real keyword-row assertions
- Script-migrations integration 68/68, knowledge projection/insert/lock/search suites 44/44, type-check, lint, and
check:auditspass
Checklist
- Code follows project style guidelines
- Self-reviewed my changes
- Tests added/updated and passing
- No new warnings introduced
- I confirm that I have read and agree to the terms outlined in the Contributor License Agreement (CLA)