Ingestion hardening: Retry route and heartbeat (part a)

Merged
BI
23d ago
+419-113

Before: a single missed worker check-in threw away a whole job's paid work. The admin Retry button used the older, less careful recovery step. A job stranded on its final attempt could hide behind old failed jobs in the health probe.

After:

  • ING-A-03 The heartbeat tolerates transient failures for a set tolerance (a third of the stale window). Definite rejections still revoke at once. startIngestionHeartbeat takes the job lease and refuses to start unless the tolerance is positive and strictly shorter than it (owner decision, 21 Oct 14:42), so an attempt always stands down before another worker can reclaim the job. Tests cover this.
  • DB-01 The Retry route calls the hardened recover_ingestion_job path. Both of its job reads join documents!inner on the owner and are declared in the derived-query inventory. Refusals are clear 404/409 responses.
  • ING-A-02 reindex-health counts jobs stranded on their final attempt directly. The delete refusal no longer tells the admin to stop a worker that does not exist.
  • EVAL-09 reindex-health counts indexed documents with no chunks and chunks with no embedding, and lists any non-zero count under integrityProblems.
  • ING-A-04 The re-index runbook now says the eval gate is advisory and done by hand until the harness is applied.

No supabase/** change.

RAG impact: no behaviour change. Retrieval, ranking and answers are untouched; this is worker lease handling, the Retry route and ops reporting.

Clinical Governance Preflight: no clinical content, cautions, thresholds or refusal logic in this PR (the OCR caution moved to #3715).

Evidence (local, offline): focused Vitest passes for worker-ingestion-attempt, worker-shadow-extraction, private-access-routes, worker-admin-source-runtime, ingestion, source-governance, table-facts and table-fact-ranking. Typecheck is clean, and the owner-scope check passes both phases. The new reindex-health counts have no unit test, because the script reads the live database. Hosted CI has not yet been observed.

Left out:

  • ING-A-08: low confidence, large.
  • ING-B-11: product decision, and the scanner is outside this repo.
  • ING-A-06: needs a new fenced cancel RPC.
  • The caption-cache key: needs a DB constraint change.
  • Autopilot alerts on the new integrity counts: these wait for #3694, which rewrites that function.

šŸ¤– Generated with Claude Code

https://claude.ai/code/session_011qMFt8SZBmLuAmkH6ko6nd

Split (owner-approved, 21 Oct 2026): this PR is now part (a), the Retry route plus heartbeat (with the stranded-job and integrity reporting). Part (b), the OCR caution, scanner warning, table-fact label carry-forward (ING-B-09) and deleted-document titles, moved to #3715. The table-row grouping (RAG-A-02) moved to #3711.