#3628 fix(rag,worker): telemetry, document attribution, comparator parity, and atomic claim scope

User description

Summary

• Persist bounded generation-degradation scalars even when the public payload is unserved, preserving the telemetry implementation already merged on main.
• Measure remaining route budget from the actual deadline at completion. Treat “no more than” and “at most” equivalently for unitless, clinical-unit, and scientific quantities.
• Keep document lookup support, citations, and derived source panels aligned. Match qualifiers across each document’s chunks, exclude partial matches, and reserve a citation for each named document.
• Render TXT/PDF viewer labels accurately and test danger-source suppression in SSE evidence previews.
• Enforce worker document/MIME allowlists inside a new atomic claim RPC overload. Keep the existing unscoped RPC for older workers; document scope configuration and fail-closed rollout behavior.
• Resolve real conflicts with main without rewriting the PR history.

Areas touched: Answer engine, App experience, Source intake and indexing, Data platform and access

RAG impact: no retrieval behaviour change — fixes affect answer verification, document-list attribution, source-panel consistency, and bounded telemetry; retrieval scores and ordering are unchanged.

Verification

• Targeted regression suites: 9 passed; 295 passed (295).
• Offline RAG gate: 26 passed; 982 passed (982); fixture validation passed 36 golden cases, 26 suites, and 26 programme cases.
• Local SQL execution: 10 scoped-claim scenarios passed, including mixed queues, intersecting allowlists, eligibility changes, active sibling leases, and locked-document MIME rechecks.
• TypeScript compilation passed with zero errors after correcting a new test fixture.
• Changed-file lint and TypeScript push guards passed. Repository-wide formatting completed. Migration-role, SECURITY DEFINER grants, organisation, diff-integrity, and maintainability checks passed.
• After helper extraction, lookup/adversarial suites passed 2 files; 31 tests.
• Independent standards/specification review completed; follow-up review confirmed citation-budget and retained-source corrections.
• Expected in CI for this event/scope: PR policy, static checks, safety, coverage, ingestion SAST, build/container checks, applicable browser/Lighthouse jobs, migration replay, and the PR required aggregate.
• Hosted CI observed at faf351d7f195a3c1bc89e1819297899a8c7f2f95: production build, safety/config, ingestion SAST, Playwright build, Lighthouse, migration application, and database behavior tests passed. The job regenerated a fresh drift-manifest artifact.
• Required CI is not green: static checks and one unit partition report stale numeric-test governance evidence and the old drift manifest. A narrowly scoped engineering drift exception is committed locally and its 43 focused governance tests pass; clinical approvals and reviewed digests remain unchanged.
• Blocked: retrieving the generated drift-manifest artifact is denied by the current workspace network policy. The manifest and final governance correction still need publication followed by a fresh exact-head CI run. Container verification was still running at the last observation.
• Local image unpack failed with no space left on device; the documented local drift push override was used to reach CI without weakening required hosted gates.
• UI verification not run: no further UI edits were made during this review; fresh browser proof for the PR’s existing label changes remains assigned to the applicable hosted CI jobs.
• Provider-backed clinical/RAG evaluations and live database checks were not run. Local offline proof does not establish clinical release readiness.

Risk and rollout

• Risk: Medium. The claim fix adds a database function overload; answer-side matching intentionally excludes partial document matches.
• Rollback: Revert the PR’s application changes. Any live database rollback requires its own specifically approved migration.
• Merge gate: This PR now includes supabase/migrations/20261017175448_scope_ingestion_claim_atomically.sql. Merging to main automatically applies it to the live clinical database and deploys the app/worker. Merge only inside the owner’s specifically approved database-change window. Do not enable auto-merge for this migration PR.
• Production effects: No live mutation or deployment was performed during this review. Scoped workers require the new overload; a missing overload fails closed instead of claiming unfiltered jobs.

Clinical Governance Preflight

Synthetic evidence was used for regression tests and local SQL execution. No patient data or live clinical sources were changed. Numeric/source safeguards remain active, and retrieval scores/order remain unchanged. The new SECURITY DEFINER overload retains service-role-only execution with PUBLIC, anon, and authenticated access revoked. Local checks do not establish clinical sign-off or authorize the database merge.

Notes

• The existing three-argument claim contract remains available during rollout.
• RAG hotspot files remain within their enforced size budgets.
• Review-thread replies and resolution remain open; this task did not explicitly authorize those GitHub conversation writes.

Summary by CodeRabbit

• New Features
  • Document previews and actions now display file types based on the document’s format, including when its filename extension is misleading.
  • Document searches can return files that collectively match the requested details, with citations tied to the matching documents.
  • Worker configurations can restrict which document IDs and file types are eligible for processing.
• Bug Fixes
  • Answer checks now recognize “no more than” and “at most” as equivalent upper limits, including in clinical quantities and alcohol recommendations.
  • Worker processing now skips jobs that are not yet due or have not reached an expired lease.

CodeAnt-AI Description

Align document matches, ingestion scope, and file labels with actual evidence and file types

What Changed

• Document lookup lists only documents whose evidence covers the requested topics, and citations and source panels reflect the same matches within the five-citation limit.
• Workers can restrict claims by document and file type before jobs are leased. Invalid allowlists claim no jobs, and excluded jobs do not use attempts or block eligible jobs.
• Clinical answer checks treat “no more than” and “at most” consistently, including when checking quantities and alcohol guidance.
• Document viewer actions and previews show the file’s actual type instead of assuming every document is a PDF.
• Generation degradation details are recorded even when the related rollout is not served, with remaining route time measured at completion. Evidence previews exclude danger-level sources.

Impact

✅ Document matches cite evidence for all requested topics
✅ Scoped workers leave unrelated jobs untouched
✅ Accurate TXT, image, and document download labels

@codeant-ai ask: Your question here

@codeant-ai ask: Can you suggest a safer alternative to storing this secret?

@codeant-ai: Your feedback here

@codeant-ai: Do not flag unused imports.

@codeant-ai: review