Library pipeline hardening: rollback restores generation details, extraction provenance, real-reader hostile tests

Merged
BI
23d ago
+1434-63

Requested by Josh · project thread

Contains a database migration. Merge only after the owner's own yes, by the owner pressing Merge. Keep this PR in draft until then.

Summary

  • #3732: index rollback restores generation details. New migration 20261021170000_restore_generation_metadata_on_rollback.sql:

    • Adds the document_index_generation_snapshots table. It is service-role only, with RLS on and no client grants.
    • Two triggers record each committed generation's details:
      • the worker-owned documents.metadata keys, which are exactly the patch_ingestion_document_state allow-list
      • the document_index_quality row
    • Neither trigger changes any commit function.
    • rollback_document_index_generation now restores those details together with the pointer. Owner-edited metadata, such as the title, is left alone.
    • A rollback is refused with target_snapshot_missing when the target generation predates the snapshots, so it fails closed.
    • The --i-understand-metadata-not-restored guard in reindex:rollback is retired, and the runbook is updated.

  • #2F8K1V: extraction provenance. documents.metadata.extraction_provenance now also records:

    • the chunker versions of the committed chunks
    • the OCR engine versions
    • the count of OCR'd pages

    This adds to the reader block already recorded since #3632. The new npm run extraction:provenance is read-only and lists documents by reader, chunker or OCR version. No migration is needed.

  • #7D8WZ2: hostile-document test. tests/hostile-document-real-extractor.test.ts runs generated hostile PDFs through upload admission, the real Python reader, the real chunker, the evidence ranker and the fenced answer source block. The PDFs are password-protected, truncated, not-a-PDF, blank, or prompt-injected with forged fences and hidden text. The live upload-to-answer leg is re-scoped to staging in the ledger.

  • #EG02B7: storage clean-up. This was already fixed by #3677 (migration 20261019155200), so a ledger done request is queued.

Areas touched: Data platform and access, Source intake and indexing, Knowledge and records, Delivery and assurance

RAG impact: no retrieval behaviour change. The worker change only adds metadata keys at commit time. Page text, chunk text, embeddings and ranking are untouched, and nothing triggers chunking or re-indexing.

Verification

  • Migration SQL was exercised against a scratch Postgres 16 in this order:

    1. commit B over A
    2. rollback B to A
    3. roll forward A to B
    4. rollback to a target with no snapshot
    5. commit C pruning old snapshots
    6. a malformed pointer value

    The results: details were restored exactly, the owner title was preserved, the no-snapshot rollback returned target_snapshot_missing, and the malformed pointer did not block the update.

  • npm run drift:manifest replayed the full schema.sql projection on the pinned Supabase Postgres 17 image: "Replay complete in 29s".

  • npm run migrations:seal sealed the new migration only. check:migration-role passed. check:function-grants passed: "all 141 SECURITY DEFINER public function(s) are revoked".

  • New and changed tests: tests/rollback-generation-snapshots.test.ts, tests/reindex-rollback.test.ts, tests/extraction-provenance*.test.ts, tests/hostile-document-real-extractor.test.ts (run with PyMuPDF 1.28.0, as in CI). All pass.

  • npm run typecheck exit 0.

  • npm run verify:cheap: "Tests 3 failed | 33288 passed". The same 3 tests fail on unmodified origin/main in this container: rag-plan-package-parity (needs origin/main checkout) and two test-runner-safety Playwright-lease tests. They are environmental and not caused by this diff.

Risk and rollout

  • Risk: the new triggers run on every documents metadata update and document_index_quality update. The documents trigger does nothing unless the generation pointer changes. The table holds at most two small rows per document.
  • Rollback: a new forward migration that drops the two triggers and restores the 20261021150000 function body.
  • Provider or production effects: merging applies the migration to the live database within seconds. Merge only after the owner's own yes. No provider calls were made while building this.

Notes

  • Existing documents have no snapshots until their next commit, so reindex:rollback --apply refuses them by design until then.
  • Ledger changes are queued requests under docs/outstanding-issues-inbox/.

Generated by Claude Code

Summary by cubic

Contains a database migration. Merge only after the owner's explicit yes; keep in draft until then.

Index rollbacks now restore the target generation's worker-owned metadata and index quality alongside the generation pointer, instead of leaving them describing the newer generation. The PR also records chunker and OCR versions in extraction provenance and runs the hostile-document matrix through the real PDF reader. No retrieval behavior changes — only metadata keys are added at commit time.

Rollback restores generation details (#3732)

  • Snapshots the worker-owned documents.metadata keys and the document_index_quality row per committed generation into a new service-role-only table via two triggers; commit functions are unchanged.
  • rollback_document_index_generation restores both, refuses with target_snapshot_missing when the target generation has no snapshot, and never touches owner-edited metadata like the title.
  • reindex:rollback reports the target_snapshot_missing refusal in its dry run so apply holds no surprise, retires the --i-understand-metadata-not-restored guard, and the runbook is updated.
  • The SQL behavior test now commits generations as the worker does and asserts the rollback and roll-forward restore each generation's details while keeping owner metadata.

Extraction provenance and real-reader hostile tests (#2F8K1V, #7D8WZ2)

  • Extraction provenance now records chunker and OCR engine versions plus OCR'd page count; the read-only npm run extraction:provenance report lists documents by version using id keyset paging, with a not recorded filter that matches its counts.
  • New hostile-document tests generate PDFs (password-protected, truncated, not-a-PDF, blank, prompt-injected with forged fences and hidden text) and run them through upload admission, the real Python reader, chunker, evidence ranker, and fenced answer source block offline, asserting forged fences are escaped and hidden instructions reach the model only inside an unbroken source fence.

Summary by CodeRabbit

  • New Features
    • Added extraction provenance reporting, including reader, chunker, OCR engine, and page-count details.
    • Index rollback now restores worker-managed metadata and quality information when snapshots are available.
  • Bug Fixes
    • Rollbacks are refused when required generation snapshots are missing, preventing incomplete restoration.
    • Improved handling and validation of hostile, malformed, password-protected, and blank PDFs.
  • Documentation
    • Updated schema inventory and rollback guidance.
    • Documented provenance tracking, rollback behavior, and outstanding operational limitations.