PR #3546 - BigSimmo/PsychSift

Review round, 16 Oct 2026 (after PR #3566)

Qodo bot commits undone. d712a154b and 7d5ea7454 are reverted with the owner's approval. Gitar confirmed both findings closed.
Main merged in. Main's 20261015181500 seal entry is added next to this PR's two entries, and no existing entry is edited (owner's named yes). Both .gitleaksignore notes are kept. The drift manifest points at the merged schema.sql, and CI's freshness job regenerates the snapshot.
Stuck job on its last try (Codex P1, 10c7b1d44). Before: a database rejection on a job's final attempt cancelled the attempt and skipped fail_or_retry, and the claim query never reclaims an exhausted job, so it stayed processing for good. After: an error carrying a Postgres SQLSTATE rolled back and left the lease held, so it reaches fail_or_retry. Connection (08), operator-intervention (57P), PostgREST gateway and code-less errors still cancel as uncertain.
Deep-memory supersede delete (CodeRabbit, major, 1263ccdf5). The card and index-unit deletes in 20261015185000 now use the same guard as the section delete, so the database repair job's stopgap rows survive a deep-memory commit. This is an unmerged migration of this PR, edited with the owner's named yes. Its seal entry and schema.sql match.
CodeRabbit quick fixes (88835edff).
- A recovered ambiguous upload writes its audit row. The replay-conflict winner path is unchanged, because the winning request already wrote its own row.
- Managed deep-memory preparation stamps rag-deep-memory-v2.
- The Storage duplicate response (HTTP 400, statusCode 409) is recognised.
- A failed egress authority load no longer records a terminal refusal for the whole attempt.
Deferred to a follow-up (ledger inbox 6aec8cab):
- Per-sheet XLSX shared-string projection. The pilot accepts only TXT and PDF.
- The pilot-client browser proof rerun.
- The live canary pair for the extractor, chunking and table-fact input changes and the original-section eligibility. This is a paid run that needs the owner's separate yes, planned right after merge.

RAG impact: no ranking, selection or ordering change. The deep-memory version stamp aligns managed output with the current rag-deep-memory-v2 that deep-memory.ts already writes. The supersede-delete guard only narrows which legacy rows are removed.

Local evidence on 67d98d762: verify:cheap passed lint and typecheck. The unit suite ran 31,489 tests, with 2 skipped. The 2 failures were a stale source-shape assertion and a shifted doc anchor, both fixed in 67d98d762, and those two files now pass 283/283. The full suite was not re-run after that fix. Migration immutability and migration-role guards pass.

Main merged in again (16 Oct 2026, after PR #3571)

PR #3571 changed two files this PR also changes. Both sides are kept.

src/lib/openai.ts: this PR's ingestion egress wrapper stays. Main's per-call embedding timeoutMs and maxRetries now pass through it, and IngestionOpenAIOptions carries both fields.
worker/python/extract_pdf_assets.py: main rewrote edge-strip row recovery around text line segments. This PR's rotated-page fix is carried into the new _extract_page_line_segments, which maps text positions into the rotated page space that table rects use. On a 90-degree page the segment matches this PR's earlier rotated block exactly. Table payloads keep both repeated_populated_geometry and main's column_ranges.
tests/pdf-table-extraction.test.ts imports pymupdf instead of the deprecated fitz alias, whose stdout warning broke the test's JSON parse.
The design-system route count is now 226, because both sides added one route.
Together the two sides put src/lib/rag/rag.ts 29 lines over its no-growth budget. The published-original layout loader this PR added moves unchanged into src/lib/rag/rag-published-original-loader.ts, and the caller signal is still read when the loader runs. rag.ts is now 4,360 lines.

RAG impact: no retrieval behaviour change. The extractor merge keeps both sides' behaviour, the loader move is a pure code move, and no retrieval, selection or ordering code changed.

Local evidence on the merged head: verify:cheap passes lint, typecheck and 31,472 unit tests (the 2 earlier failures were the fitz warning, now fixed). The Python extractor tests pass 8/8 with PyMuPDF 1.28.2, and the table-extraction tests pass 3/3.

Import loops fixed (16 Oct 2026)

Before: the full unit suite failed its architecture check because of two runtime import loops this PR introduced. One was between the XLSX preflight and provenance modules, and the other came from worker/candidate-chunks.ts re-exporting the scoped candidate projection.

After: the XLSX schemas and classifiers move unchanged into src/lib/extractors/xlsx-source-model.ts. Provenance re-exports them, so every existing import still resolves. The candidate-chunks re-export is removed, and its one test imports directly. The clinical doc line anchors that this PR's earlier growth of rag-source-block.ts and rag.ts had shifted are remapped. The owner gave a named yes for the protected extractor edit (docs/decisions/2026-10-16-import-loops-fixed-in-3546.md), and the ledger item that queued the loops separately is removed.

RAG impact: no retrieval behaviour change. The extractor edit is a pure code move with identical exports, and no ranking, selection or ordering code changed.

Codex reference-branch fixes (16 Oct 2026)

Before:

A plain-text admin source with no saved section layout was refused.
A claimed job row that failed the shape contract was left until its lease expired.
If two search methods found the same passage and only one carried its section path, a single-document summary failed with an error.

After:

A plain-text source may omit its layout. A layout that is present must still match the source hash.
A malformed claimed row is failed terminally through the existing fenced fail_or_retry_ingestion_job, but only when its claim identity parses into a current, unexpired fence owned by this worker. Payload fields are never used, and a failed transition never strands valid siblings.
A repeat sighting with less detail keeps the fuller receipt. A genuine conflict still fails closed (owner's named yes; docs/decisions/2026-10-16-duplicate-sighting-receipt-merge.md).

These are ported from codex/pr3546-f04-candidate (f04c841c4). The owner chose to put the native PDF provenance change and the pre-claim refusal into a follow-up PR, and both are queued in the ledger inbox. No new migration.

Owner decisions applied (16 Oct 2026)

Before:

Two upload tests expected an admitted document to be deleted when the upload was cancelled late.
A broad question that mixed sources returned an error.
Every signed-in answer skipped the saved-answer and search caches.
The privacy readiness table still listed the OpenAI and Railway agreements as pending.

After:

An admitted upload survives a late cancel. The two tests are rewritten, and cleanup before admission is still covered.
A mixed original-section selection falls back to a normal cited answer. An integrity failure inside one published document still fails closed.
Only an unscoped broad summary bypasses the caches.
The readiness table now matches the register.
src/lib/deep-memory-preparation.ts is added to ragRankingPatterns.
The import cycles are now fixed in this PR (see above).
The indexing repair job keeps provider calls off until its scanner is connected.
Each decision has a docs/decisions/2026-10-16-* entry.

Pilot ticket: the blanket refusal of All-documents answers on a stale ticket (2d05bf031) is reverted. A stale ticket now judges those requests exactly as a valid ticket does. It still refuses the pilot owner, pilot document and pilot query, and an unreadable ticket still refuses everything.

RAG impact: no ranking change. Original-section delivery is limited to the query class that can use it, and it falls back to baseline packing on mixed selections. When one row is observed twice, the receipt merge now keeps the fuller receipt. Selection and ordering are untouched. No canary run.

Review fixes (15 Oct 2026, Claude review thread)

Before: the worker called five database functions that did not exist, so a real upload or reindex would stall in "processing" and deep memory would fail silently. An expired pilot ticket would return 503 to every Answer request, and 74 offline unit tests failed.

After: the missing fenced functions exist in new migrations, and ordinary Answers keep working when a pilot ticket is stale.

Database: this PR adds two migrations.

20261015185000_fenced_ingestion_reset_and_deep_memory.sql adds the fenced reset, image upload and deep-memory functions.
20261015200000_fenced_ingestion_followups.sql adds a null-safe staging ownership check and allows extraction_provenance in the document patch.

Every function is SECURITY DEFINER with an empty search_path and is callable by service_role only. Each checks the lease before any write and never touches the served generation. Both migrations are sealed. Merging this PR applies both migrations to the live database within seconds. The owner gave a named yes to merge on 16 Oct 2026.

Bot commits reviewed (Qodo, 18:36):

Kept the upload log context.
Kept requireEveryPage: false, because the Python extractor emits no page provenance.
Kept store: false under the pilot.
Kept the alcohol-summary gate.
Undid four changes: the inverted admin source check, the wrong reset RPC, the sparse-table admission, and the whole-chunk negation filter.

Other fixes:

The upload user client now comes from the cookie session only.
A 409 replay conflict uses the standard error envelope.
/admin/pilot is linked from the admin ingestion panel.
allowImportingTsExtensions replaces six suppression comments.

Summary

Connect administrator ingestion and fenced generation persistence to source-grounded Answer delivery. Changes from main are preserved, including its citation-tap, keyword-timeout, owner-permission and SQL-behaviour CI changes. PR #3544 is already merged and is preserved through ancestry. The native PDF adapter retains page provenance, intact text and table/image relationships. The complete qualified history is recorded in docs/audit/ingestion-rag-2026-10-15/complete-record.md.

Areas touched: source-intake, answer-engine, data-platform, app-experience, delivery, knowledge.

Verification (earlier, by the original author)

Hosted pilot: exactly one unchanged approved TXT question and one Copy. An independent clinical and privacy review accepted the output. Low trust and five warnings remain visible.
The corrected recorded-response proof passed 13 required checks and 10 unsafe cases. The historical at most safe-refusal limitation still fails; no assertion was weakened.
Bounded OCR repair: OCR now requests 450 DPI within the unchanged limits. This is local evidence only, not hosted acceptance.

Risk and rollout

This PR contains migrations, so merging applies them to the live database. The owner gave a named yes to merge on 16 Oct 2026. After merge, the post-merge live-drift workflow must pass both check:drift and check:migration-history.
Worker queue dispatch remains disabled. Bulk ingestion and re-indexing have not started.
General scanned-table admission remains held.
Generic legal/clinical release GO and all-user corpus-wide serving are not inferred.

Clinical Governance Preflight

Source, numeric, period, population, citation, fencing and server-only access checks retained.
Actual bounded clinical delivery/Copy and independent specialist review accepted.
Existing restricted reference/education owner decisions and contractual facts retained.
General product legal/clinical classification and unrestricted release readiness established.

🤖 Generated with Claude Code

https://claude.ai/code/session_01M6gRtN6r4vZDgDfzMtjs95