#8330 improvement(knowledge): scan processing recovery per connector and prune finished outbox events

Summary

• Processing recovery discovered candidates by walking the global (uploaded_at, id) recovery index, joining each document to its knowledge base and connector, and rejecting it only afterwards. Retained inputs of paused or dormant sources sit at the front of that order, so every call read through all of them first. Follow-up batches (NOT IN attempted connectors) walked the same range again
• Discovery now starts from the eligible connectors and reads each one's oldest candidates through a new partial index doc_connector_processing_recovery_idx (connector_id, uploaded_at, id) with a CROSS JOIN LATERAL, then takes the oldest overall. Paused sources cost nothing. Batch size, the follow-up batches for refused connectors, blocked knowledge bases, the lock-time recheck, and liveness are unchanged, so recovery picks the same candidates as before
• The old doc_processing_recovery_idx stays for now and is dropped in a later contract migration
• Outbox retention: outbox_event never deleted completed rows. The outbox processor now prunes completed rows older than 7 days in bounded, SKIP LOCKED batches, only for knowledge.document.processing.recover and knowledge.document.storage.cleanup. Both have random ids, and nothing reads their completed rows. Every other event type is kept, including idempotency-keyed ones and the checkpoint expiry events, whose completed rows are read back by id, as are pending, processing, and dead-letter rows

Type of Change

• Improvement

Testing

• stored-document-recovery.integration.ts:
  • with 5,000 older paused-source documents ahead of a recoverable one, discovery stays under 1,000 buffers (EXPLAIN ANALYZE BUFFERS) and recovers it in one call. The old query read 10,516 buffers
  • a connector whose full batch is refused doesn't block a sibling's candidate in the same call
• retention.integration.ts: deletes only old completed allowlisted rows and drains past one batch. Removing the status, age, or type filter, or the batch loop, turns it red
• Migration replayed twice (concurrent build, index valid); check:migrations origin/staging passes
• Type-check, lint, check:audits, and the recovery, outbox, and document unit suites pass

Checklist

• Code follows project style guidelines
• Self-reviewed my changes
• Tests added/updated and passing
• No new warnings introduced
• I confirm that I have read and agree to the terms outlined in the Contributor License Agreement (CLA)