#3596 fix(rag,ingestion): harden generation isolation, cache fail-closed stamps, summary verification, and edge/worker alignment

User description

Summary

• Harden RAG retrieval, document summary context, and related-document enrichment against uncommitted or superseded index generations:
  • Enforce committed-generation filtering in fetchBestDocumentLookupChunks, loadChunksForMemoryCards, loadChunksForSignalMatches, loadDocumentSummaryContext, summarizeDocument, GET /api/documents, and fetchRelatedDocumentMetadata.
  • Include public-corpus (owner_id IS NULL) rows when attaching index quality metadata for authenticated users (attachIndexQualityMetadata) while stripping internal owner_id fields before caching.
  • Apply applyNumericVerification(assessAndEnforceClaimSupport(answer), results) to summarizeDocument, preserve numeric-band coherence conflict warnings in conflictsOrGaps, clear bestSource / supportedClaims / evidenceAssessments when verification refuses an ungrounded summary, allow signed-in users to summarize public documents, and guard against non-indexed documents.
• Harden RAG cache fail-closed behavior and tenant invalidation scoping:
  • Enforce isRagCacheAccessAllowed in answerCacheLookupAllowedForRequest and shadow-mode guards on search cache lookups.
  • Fail closed on :index-stamp-unavailable across local and shared search/answer cache reads and writes without evicting valid local cache entries on transient DB hiccups or caching :index-stamp-unavailable in cacheIndexingVersionCache.
  • Add invalidation-epoch fencing (captureInvalidationEpoch / invalidationEpochChanged) across setCachedSearch.
  • Pass { affectsPublicCorpus: !job.documents.owner_id } in worker/main.ts so private document completions do not wipe the shared public/anonymous RAG cache, and avoid duplicate anonymous shared-cache deletes when ownerId === "anonymous".
• Harden ingestion queue rollback, quality diagnostics, chunker table matching, and indexing-v3-agent edge function alignment:
  • Accept both CHUNKER_VERSION ("rag-chunker-v1") and DOCUMENT_CHUNKER_VERSION ("rag-chunker-v2") in matchesNativeTableBlock.
  • Deduplicate ingestion_job_stages by stage_name and scope failedJob / failedOcrStage in GET /api/ingestion/quality so historical or superseded OCR failures do not trigger false-positive danger alerts after a completed retry.
  • Roll back non-atomic documents updates under the updated_at fence when enqueueDocumentReindexJob loses a 23505 unique-constraint race to an active "processing" job.
  • Order document_admission_events deterministically in indexing-v3-agent, populate source_chunk_id: payload.source_chunk_ids[0] ?? null in upsertCoreEmbeddingFields to satisfy assert_ingestion_artifact_sources, and accept local-worker rag-deep-memory-v2 structured summaries in normalizeSourceAnchoredSummary.
  • Coerce blank string env overrides ("") for RAILWAY_WEBHOOK_SECRET and SUPABASE_INGESTION_WEBHOOK_SECRET to undefined in src/lib/env.ts.

Areas touched: Source intake and indexing, Answer engine, Data platform and access
RAG impact: no retrieval behaviour change — enforces fail-closed committed-generation isolation on fallback/fast-path chunk loaders, cache stamp error guards, and summary numeric verification without altering committed-corpus ranking weights, scoring formulas, or retrieval candidate selection order

Verification

Paste the decisive line from whatever you ran — exit code 0 alone is not proof.

• Checks run:
  • npm run typecheck:internal -> node --max-old-space-size=8192 ./node_modules/typescript/bin/tsc -p tsconfig.typecheck.json --noEmit (0 errors)
  • eslint on all 20 modified files -> 0 errors, 0 warnings
  • npm run check:diff-integrity -> [diff-integrity] PASS — 8 changed test file(s), 122 -> 130 test case(s), against base a5b118f95.
  • npm run check:migration-immutability -> Migration immutability guard passed: 316 shipped migrations unchanged.
  • npm run check:owner-scope -> ✓ owner-scope phase 1: 127 src/app/api files clean against 56 owner-scoped tables. ✓ owner-scope phase 2: 172 direct, 32 user-keyed, 29 derived-tier, 2 untiered-table and 1 dynamic-table-dispatch queries scoped on their chain or declared; 8 versioned-RPC call sites, all literal.
  • npm run check:rag:fixtures -> Offline RAG fixture and manifest validation passed (36 golden cases, 26 suites, 26 programme cases).
  • npm run eval:rag:offline -> Test Files 26 passed (26) | Tests 979 passed (979) | Offline RAG fixture and production-contract checks passed.
  • npm run eval:rag:adversarial:offline -> Adversarial fixture contract passed (24 synthetic cases, 8 categories, 6 canaries). | Test Files 1 passed (1) | Tests 25 passed (25) | Offline adversarial fixture validation and regression harness passed.
  • Focused Vitest suites (tests/rag-shared-cache.test.ts, tests/rag-cache-invalidation.test.ts, tests/rag-document-summary.test.ts, tests/document-enrichment.test.ts, tests/chunking.test.ts, tests/indexing-v3-agent.test.ts, tests/ingestion-quality-route.test.ts, tests/private-access-routes.test.ts) -> 8 passed (8), 254 passed (254)
• Verification not run: eval:retrieval:quality, eval:rag, and eval:quality (provider-backed live evaluations are owner-run and approval-gated; no committed-corpus ranking weights or ordering rules were modified).

Clinical Governance Preflight

• Source-backed claims still require linked source verification before clinical use
• No patient-identifiable document workflow was introduced or expanded without explicit governance approval
• Supabase target remains PsychSift Production (sjrfecxgysukkwxsowpy)
• Service-role keys and private document access remain server-only
• Demo/synthetic content remains clearly separated from real clinical sources
• Source metadata, review status, and outdated/unknown-source behavior remain conservative
• Deployment classification/TGA SaMD impact was checked when clinical decision-support behavior changed

All changes strengthen conservative source grounding and generation isolation: summarizeDocument now runs applyNumericVerification(assessAndEnforceClaimSupport(answer), results) and clears bestSource / supportedClaims / evidenceAssessments on ungrounded or unsupported summaries, uncommitted/superseded index generations and hidden labels are excluded across fallback and fast-path readers, and cache stamp failures fail closed without serving unverified cached responses.

Risk and rollout

• Risk: Low. All changes tighten fail-closed guards (committed-generation matching, cache stamp unavailability handling, summary numeric verification, and ingestion stage/queue consistency) while preserving existing API response shapes and committed-corpus ranking formulas.
• Rollback: Revert the merge commit (git revert <sha>); no database schema migrations are included.
• Provider or production effects: None.

Notes

• Includes two rounds of independent adversarial review and regression tests across tests/rag-shared-cache.test.ts, tests/rag-document-summary.test.ts, tests/document-enrichment.test.ts, tests/chunking.test.ts, tests/indexing-v3-agent.test.ts, and tests/ingestion-quality-route.test.ts.

Summary by cubic

Hardens RAG retrieval, document summaries, and caches against uncommitted or superseded index generations, and fixes ingestion quality, queue rollback, and worker cache invalidation scoping.

RAG retrieval and summaries

• Filters labels, summaries, and chunks to the committed index generation across fallback and fast-path readers, including GET /api/documents and related-document metadata; public rows are filtered server-side by metadata and then projected to public fields so metadata never leaks, hidden labels are excluded, and index-quality metadata includes public-corpus rows for signed-in users.
• Runs document summaries through numeric verification and clears bestSource, supportedClaims, and evidenceAssessments on ungrounded output.
• Lets signed-in users summarize public documents, enforces indexed status before summarizing, and returns no chunks for non-indexed documents.

Cache and ingestion

• Fails closed on cache reads and writes when the indexing stamp is unavailable, without evicting valid local entries; shadow-mode cache lookups are also guarded.
• Fences search cache writes with invalidation epochs and scopes public-corpus invalidation so private document completions don't wipe the shared anonymous cache.
• Rolls back document status when a reindex job loses a unique-constraint race, restoring processing for the active job and scoping the rollback to the owning row.
• Skips superseded failed jobs and OCR stages in ingestion quality after a completed retry.
• Accepts both CHUNKER_VERSION and DOCUMENT_CHUNKER_VERSION for native table blocks, local-worker rag-deep-memory-v2 summaries, and deterministic admission-event ordering in indexing-v3-agent; blank webhook secret env overrides are coerced to undefined.

CodeAnt-AI Description

Prevent stale index data and unsupported claims from appearing in search and summaries

What Changed

• Search, document lists, related-document details, and summaries now use only data from the document’s committed index generation; hidden labels are excluded and the newest valid summary is shown.
• Document summaries require indexed evidence, verify numeric claims, and remove source recommendations when a summary is unsupported.
• Cache reads and writes stop when the current index version cannot be confirmed, preventing stale results from being served or saved.
• Ingestion reviews ignore failures from older attempts after a successful retry, and enqueue races restore the document’s prior status and counts.

Impact

✅ Fewer stale search results and document details
✅ Fewer unsupported numeric claims in summaries
✅ Accurate ingestion failure status after retries

@codeant-ai ask: Your question here

@codeant-ai ask: Can you suggest a safer alternative to storing this secret?

@codeant-ai: Your feedback here

@codeant-ai: Do not flag unused imports.

@codeant-ai: review

Summary by CodeRabbit

• Bug Fixes
  • Document summaries and search results now better reflect the latest committed document content, while excluding hidden labels and outdated summaries.
  • Ingestion quality reports are less likely to show failures from older jobs after a newer job completes.
  • Answers are checked against their sources; unsupported claims are no longer presented as grounded.
  • Search and answer caches now fail safely when document index status is unavailable, reducing the chance of serving stale results.
  • Table crops are supported across both page-based and document-based processing.
  • Ingestion status is better preserved when duplicate jobs are detected, and document updates are more carefully scoped to their owners.