feat: ingest and query organization datasets in Recoup

Merged
SI
3d ago
+859-162

Organization imports can now land in Recoup's existing database: original and processing files remain private, while validated gold becomes queryable as organization-owned dataset/import records. External client database delivery stays optional.

  • Organization creates register an import and return dataset_id; passing that ID on later creates adds an immutable import version.
  • Workers atomically store ready gold and its completion receipt through the guarded database gateway. Held/rejected results retain metadata without gold rows; lost acknowledgements and completion-marker failures remain retryable.
  • Add authenticated HTTP/MCP datasets, imports and paginated records reads with organization scope, explicit import versions and per-operation limits. GET remains read-only.
  • Replaying a completed private import can repair its shared catalog entry after integrity-checking saved gold. No per-org databases, schemas or new client credentials are required.

Depends on Database PR #133. Apply that migration to the existing Recoup project before deploying/enabling this consumer. INGESTION_ENABLED remains off pending hosted qualification.

Validation on 98fba08d: every CI check passes, including the full API regression suite, production/contract/workflow compiler checks, lint, formatting and OAuth compatibility. The ingestion job passes 107 tests on each of PostgreSQL 15 and 17; the optional joined test is skipped in that ordinary CI job and passes separately against the real migrations in the local database harness. The database harness passes twelve SQL cases plus that joined API lifecycle on PostgreSQL 17, covering exact values, acknowledgement-loss recovery, versioned imports, cross-org denial, revocation and bounded pagination. Hosted PR-preview metadata confirms the table grants, guarded RPC and page-budget correction. Vercel preview builds successfully; preview and production HTTP checks return the expected disabled 404 with private/no-store headers.

Review: missing/corrupt saved gold, invalid UTF-8, malformed completion metadata, exact redaction, denied unit membership and explicit membership fixture columns are covered. Addressed review threads have been resolved. The remaining operation-processor refactoring suggestion is retained for human review. Inline record payloads have a 2 MiB budget; oversized records remain stored and require a future large-object export path.

No production migration, intake activation or new client database was performed. See lib/ingestion/CATALOG.md for operations, shared ownership, limits, validation boundaries and rollout order.