Preserve PDF evidence and reject unsupported RAG source claims

Merged
BI
6d ago
+28047-298

Summary

PDF candidate generations now retain source-bound summaries and generated labels with their page, table and image evidence, then publish the pointer, projections and job completion atomically. A separate renderer-only path excludes the original native layer and remains held until a complete source-bound review is validated. Ordinary reindex requests keep an owner-only query; shared-library readmission requires explicit administrator authority.

Two reproduced answer-trust defects are also corrected. Generated index/synopsis/memory narrative can no longer supply direct claim proof or exact quotes. Explicitly withdrawn following recommendations are withheld from answer prose and quote cards, including numbered-list and duplicated adjacent-context cases. Original short quotes, negated withdrawal wording and later independent recommendations remain supported. Existing structured table/image provenance remains a separate limitation.

Areas touched: Source intake and indexing, Answer engine, Data platform and access, Knowledge and records, App experience, Delivery and assurance.
RAG impact: behaviour change — the latest approved local current-main native gates passed, but its strict financial wrapper is incomplete; the main-only canary pair remains incomplete. Retrieval/ranking haystacks are unchanged; ingestion evidence and answer-proof eligibility change.

Required check change approved: The owner's direct 5 November 2026 reply was "Approve the CI OCR setup change", recorded in the task checkpoint. The workflow change installs Tesseract and pinned Pillow/pytesseract for existing unit jobs. It preserves all tests, checks and thresholds and grants no production authority.

Verification

Published head 004d6d357d88a9eab223e3e97c39a44f01fa7c26 adds a diagnostic-only correction and retains the checksum-verified original and additive bundles, normal integration merges, and cloud follow-up 2626b4a4db. The original checkout's unpublished changes remain intact.

  • New diagnostic fix: private source dumps retain the existing context relevance verdict through a four-value allowlist or null, with no reclassification or free-text explanations. The new regression failed before the fix, then 26/26 tests passed. Fresh TypeScript, scoped lint, installed-tree integrity, documentation maintenance, formatting and diff checks pass; independent technical review found no blocking issue. The earlier 7 suites / 585 tests, strict hazard and both owner-scope results remain reused evidence for unchanged source-trust inputs. Human assurance dates, states, risks and approvals remain unchanged.
  • Reused evidence on unchanged inputs: clean 355-migration / 20-SQL-file replay, chain/mirror parity and identical manifest regeneration; 21 rendered + 15 native Python tests; earlier focused pipeline/tenancy/readmission checks. No applied migration, production drift allowlist or test threshold changed.
  • Current-head CI 42111556501 completed successfully for exact head 004d6d357d88a9eab223e3e97c39a44f01fa7c26: 17 selected jobs passed, six skipped, zero failed. Both unit partitions/coverage, all browser shards, Lighthouse, builds, static/safety, container verification and migration replay passed. Source contracts, secret scans, platform smoke, PR policy and mergeability also passed. The unchanged-base CI 42066823624 at 45ba6b48fb1c71fb42886c237a6327888e14e102 completed successfully. App/Playwright builds, static/safety checks, both unit partitions and combined coverage, authoritative migration replay, ingestion SAST, container build/runtime/security/evidence, Lighthouse, critical journeys and all three broader browser shards pass. Prior cloud-head CI 42061052713 passed Lighthouse, both app builds, container verification, migration replay and unit partition 1. Its static failure and six unit failures all identified the stale helper evidence digest now corrected. Browser jobs were skipped; those are not passing evidence. Earlier performance failures remain historical; no baseline or budget was weakened.
  • The owner approved the exact private two-page figure review record. Real fresh inspection plus strict source/review binding validation passed: 2 pages / 2 tables / 0 diagrams. Private originals and the earlier eight-page fidelity pack remain preserved outside Git; fresh inspection artifacts were cleaned. The other two sources' irregular merged-table relationships remain held. This validates structure, not reviewer authentication, clinical acceptance or a live policy grant.
  • The bounded Sol comparison is complete and remains accepted:false: strict scores 9/12 candidate and 8/12 control; estimated US$0.3151065, not an invoice. Diagnostic limitations and latency failures are retained. No model activation occurred. The separate approved local canary used existing control models. The main-only protocol still requires a current-main green baseline and, after an approved owner merge, the reserved post-run and artifact comparison.
  • Historical local baseline on exact main a64d237eaed20c046d49a75cae000355c8a5dd8e was not accepted. All 36 golden retrieval cases passed with document/content recall 1.0. The native 44-answer blocking threshold is unsupported-route p95 5,924 ms above 4,000 ms; separately one case missed an expected source. The five-case registry omission is retained and disclosed. The closed financial wrapper also reports incomplete, retaining five uncertain embedding reservations; all six answer-generation attempts settled. Existing artifacts do not attribute the delay or locate the admission source before selection. No threshold, alias, source expectation or model setting was changed. The owner subsequently approved the exact two-case diagnostic extension. It completed within 23.9 seconds with two embeddings/155 tokens, two free counts, zero generations and no unsettled attempts. The source miss reproduced; the unsupported case passed at 1,530 ms this time. Three synthetic replays show that partial relevance can reproduce the rank-seven order, while direct controls keep the expected source in the top five. Existing tests accept the alternative bound fact pair actually cited. Missing live verdicts prevent prescribing a ranking or alias change; the diagnostic fix preserves that annotation for future authorized captures. At that checkpoint three of four approved batches were consumed; the post-merge batch stayed reserved. The later bounded recheck is recorded below. Local execution is not hosted EvalCanary acceptance.
  • Fresh bounded baseline recheck: native gates passed; financial wrapper incomplete; complete release baseline remains unaccepted. The direct owner continuation authorized one additional US$3/30-minute baseline within the unchanged US$9 total, increasing the batch limit from four to five and retaining the US$3 post-merge reservation. Independent review cleared exact launcher hashes, original approvals and the prior liability before execution on unchanged main a64d237eaed20c046d49a75cae000355c8a5dd8e. The run completed in 189.8 seconds: 36/36 golden cases passed with document/content recall 1.0; the identical 44-answer cohort had no native blocking threshold failures, with unsupported-route p95 1,254 ms against 4,000 ms. The same known admission expected-source miss remains. Of 87 paid attempts, 86 settled; one embedding lacks usage accounting, retaining its full US$0.000181 reservation. Conservative run liability is US$0.298518; cumulative local-canary liability US$0.610782, not invoice cost. All 12 uncertain attempts across the four batches remain fully reserved. The existing strict wrapper exited canary_not_complete; no settlement, acceptance override or additional automatic run occurred. Four of five batches are used; post-merge US$3 stays reserved. Independent source-history review confirmed the narrow HITH programme policy was deliberately excluded from the broader admission aliases in 5bd7e8d65f8c0640358d26c7006b79d2c48089e2; no alias, ranking or threshold change is justified by truncated input captures. Local evidence is not a hosted EvalCanary result or clinical acceptance.
  • Historical read-only production catalog checks (5 November 16:37–16:40 UTC): latest successful app and worker SHA remains c92e78113abd936c53f88e7e069b2c7331c735a2; WORKER_PAUSED=true; all six draft migration-history rows and six prepared public RPCs are absent. The catalog probe invoked no application RPC. Fresh Railway reads at 05:41 AWST on 6 November confirmed the same successful app/worker revision and worker hold. The PR stays open with auto-merge off.
  • Hosted verification remains within the approved US$5 maximum: conservative usage/rate estimate US$0.11396624 including the current head's artifacts, not an invoice. No task preview existed in the latest inventory; one approved replacement is subject to exact-resource cleanup. Production and the unrelated older preview remain preserved.

Follow-up fixes (6 November 2026, 1bf4584c)

  • Withdrawn-recommendation filter (src/lib/rag/rag-claim-support.ts):

    • The sentence splitter now recognises abbreviations, so "e.g. 400 mg..." can no longer leak the tail of a withdrawn instruction.
    • "The recommendation below has been withdrawn" now withholds the item that follows.
    • A trailing "This/the above recommendation has been withdrawn" withholds the preceding sentence or list item.
    • "has not yet been withdrawn" is no longer treated as withdrawn.
    • Six regression tests were added (30/30 in tests/rag-source-trust.test.ts), and the hazard-control digest for this sealed file was resealed.
    • This widens what is withheld from answer proof, so the eval-canary before/after pair is still required and has not been run.

  • Blank OPENAI_QUERY_CLASSIFIER_MODEL, OPENAI_SUMMARY_MODEL and OPENAI_INDEXING_MODEL values are now treated as unset in Node, matching Edge (tests/env.test.ts 10/10).

  • Readmission timing (991b8014, new draft migration 20261106184724_defer_readmission_ranking_authority.sql, owner-chosen pending-aware design):

    • Readmission now stores the 11 flat ranking fields in metadata.pending_reindex_authority, with a snapshot of the prior live values, instead of writing them live.
    • ingestion_generation_evidence.apply_pending_reindex_authority copies them live only after the generation commit and job completion succeed in complete_managed_full_pdf, complete_managed_general_pdf and complete_admin_pdf. It rolls the terminal back if the live values changed mid-attempt, for example a licence withdrawal.
    • lock_ingestion_document_authority keeps its key set and reads the pending values when the block exists, so digests for documents without the block are byte-identical.
    • The worker flat-metadata checks are pending-aware.
    • The manifest def_hash values were derived after reproducing the existing values locally. CI's Docker replay is the final proof.
    • Known limits: rolling back a generation after apply keeps the new flat values, and a readmitted document finished by another terminal keeps its pending block, so ranking keeps the old values.

  • 351b9f3d: date-relative roster SQL fixture (cherry-picked from #3902). The fixture's hard-coded November dates now fall inside the 7-day approval window and fail Migration replay on every database PR.

  • Checks: typecheck, eslint and prettier on changed files, owner-scope, migration role, immutability and function grants all pass. A focused run selected 1,876 files and its only remaining failures are environmental: 19 tests fail because pymupdf is missing in the container, or because of the npm version-pin smoke test.

  • Review threads (060c83cd): all 12 resolved. New forward migration 20261106223559_full_pdf_packet_parent_lookups.sql builds the chunk, image and section id lists once, instead of rescanning embedding-bearing arrays per row, in assert_full_pdf_packet_v1 and assert_rendered_pdf_packet. A randomised old-versus-new comparison over 30,000 cases found 0 mismatches. The full pgvector replay runs in CI.

Risk and rollout

Risk: high — privileged ingestion and answer-proof changes require the existing source/clinical, behavior and production release gates.
Rollback: retain prior active generations and keep ingestion paused; compensate database changes only through a separately reviewed owner-approved migration.

  • Privileged generation-validation SQL and answer eligibility change. New summary/render policies default off; bulk PDF 409 and source-fidelity holds remain enforced. Existing structured evidence, cross-field chronology, adaptive delivery and live answers are not universally certified.
  • Merge only after the owner's own specific yes to the eight named draft migrations (the six below plus 20261106184724_defer_readmission_ranking_authority.sql and 20261106223559_full_pdf_packet_parent_lookups.sql), successful required CI, applicable source/clinical gates and a green baseline canary on current main; the required paired post-canary follows the approved merge. The owner must press Merge manually. Auto-merge remains off. Merging applies database changes to PsychSift Production within seconds.
  • Exact targets: Supabase sjrfecxgysukkwxsowpy; Railway project 5deaad0b-675a-4c13-978e-5ca2b5b877f9, production environment 6aa16f7b-d3e8-4aa2-9854-ee9ead9fcbd4, existing app/worker services only.
  • After an approved owner merge, require post-merge live-drift with both check:drift and check:migration-history green, verify actual app/worker revision and retain WORKER_PAUSED=true, library stop and claim-disable controls. Verify new summary/reviewed-render policies remain OFF and existing model selectors remain unchanged. Do not invoke readmission as a test: it queues work. No claim, re-index or re-ingestion is authorized here.
  • Rollback retains prior active generations and the worker hold. Compensating production SQL needs a separately reviewed owner-approved change; applied migration history is not rewritten. Model selectors stay unchanged.
  • Provider effects: the OpenAI comparison is finished. The owner separately approved a US$5 hosted-verification maximum and cleanup of only this new temporary preview before handback. Standard public-repository runners are included; no production data/secrets enter the preview. The separate local canary authorization retains its run-count and spending limits; further pre-merge live work is held.

Clinical Governance Preflight

  • Source-backed claims still require linked source verification before clinical use
  • No patient-identifiable document workflow was introduced or expanded without explicit governance approval
  • Supabase target remains PsychSift Production (sjrfecxgysukkwxsowpy)
  • Service-role keys and private document access remain server-only
  • Demo/synthetic content remains clearly separated from real clinical sources
  • Source metadata, review status, and outdated/unknown-source behavior remain conservative
  • Deployment classification/TGA SaMD impact was checked when clinical decision-support behavior changed

The approved figure record is structurally validated. Current-head selected hosted CI passed. Clinical acceptance, a green current-main baseline and paired post-canary, and specific production approval remain open. Engineering review and a technical digest refresh grant no clinical sign-off.

Notes

The draft migrations are 20261103000000_preserve_wa_admission_jurisdiction.sql, 20261103000100_managed_general_native_pdf_text_terminal.sql, 20261103000200_administrator_legacy_reindex_readmission.sql, 20261103000300_managed_full_pdf_generation.sql, 20261104000000_managed_full_pdf_summary_projection.sql and 20261105000000_managed_reviewed_rendered_pdf.sql.

Transfer PR #3890 remains unchanged. The single checkpoint records source identity, scoped evidence, approvals and the later post-pilot shadow-driver requirement. No live readmission or ingestion was performed.

Summary by cubic

Preserves source-anchored PDF evidence end to end and rejects unsupported RAG proof, with production rollout still gated on owner approval.

What changed

  • PDF candidates keep source-bound summaries and labels with page, table, and image evidence; a reviewed-rendered path excludes native extraction and stays held until full source-bound review.
  • Administrator readmission checks the original file and current authority, and managed PDF images route through private, non-cacheable delivery. Ranking fields (publisher, jurisdiction, dates, document status, source role, licence policy, rights evidence) are deferred to a pending_reindex_authority block until the reindex terminal commits; a draft migration fails closed if a live flat value changes mid-attempt.
  • Reindex gate rejects invalid or impossible evidence with NO_GO; answer support excludes withdrawn recommendations, nested list items, trailing "above/previous" notices, abbreviations, and generated summaries from direct proof and quote verification.
  • Bounded gpt-6.1-sol comparison finished with activation kept separate; blank workload models are treated as unset.
  • Fixed hosted regressions (migration seals, OCR runtime prerequisites, per-chain tenancy guard, Windows process-tree termination) and added offline tools whose results do not activate changes.
  • Added a forward migration that collects full-PDF packet parent IDs once per validation instead of rescanning rows, removing quadratic validation cost without behavior change.

Rollout

  • Merging applies draft migrations to PsychSift Production; requires owner explicit approval and manual merge with auto-merge off.
  • After merge, keep WORKER_PAUSED=true, verify drift with check:drift/check:migration-history, and do not treat readmission as a test. No production re-index, ingestion, readmission, or model activation is authorized here.

Summary by CodeRabbit

  • New Features
    • Added authenticated access to retained PDF images with private, non-cacheable delivery.
    • Added administrator-authorized readmission for individual PDF reindexing, with checks on the original file and current authority.
    • Added policy-controlled processing paths for native-text, rich-content, and reviewed-rendered PDFs.
  • Improvements
    • Evaluation reports now include estimated answer-generation costs and pricing details.
    • Added offline tools to compare answer models and assess reindex readiness; results do not activate changes.
    • PDF reindexing requires readmission, and invalid evaluation data cannot pass the readiness gate.
    • Answer support checks exclude withdrawn recommendations and generated summaries as evidence.
  • Documentation
    • Updated guidance for PDF processing, reindexing, and model comparisons.

Generated by Claude Code