fix(indexing): keep one failing document from failing its batch
Description
Keeps one document that cannot be indexed from failing the 15 documents batched with it. Part of https://github.com/onyx-dot-app/onyx/issues/15521.
Why
- A Contextual Retrieval document summary call that raised (for example an input over the model's context window) propagated out of the batch, so every document in the batch was recorded as failed.
- The customer's mailbox showed one oversized email failing 16 documents on every run.
What
- A failed Contextual Retrieval call (document summary or chunk context) fails that document, which is recorded as its ConnectorFailure. Documents are never silently indexed without context.
- A batch that raises is indexed again one document at a time, and only the documents that still raise are recorded as failures.
- Only a final failure is reported to Sentry.
- A failed external document push is never retried per document, since the documents are already written and would be skipped as unchanged.
How Has This Been Tested?
Unit tests in test_indexing_pipeline.py: a failed summary or chunk-context call raises for the document, a batch with one bad document fails only that document, a single-document batch is not retried, a clean batch passes through, and a push failure fails the batch as a whole.
Live on dev1 at 642b4982e1 through the real workers, Contextual Retrieval forced on for the processing worker with its route to the model provider cut: a two-file batch, one file long enough to need a summary and one short enough to need no contextual call, ended COMPLETED_WITH_ERRORS with one error naming the long file (connection refused from the provider) while the short file indexed. Earlier on this branch, a database trigger that failed one document of a six-document Outlook batch produced one error for that document and five indexed documents.
Additional Options
- [Optional] Please cherry-pick this PR to the latest release version.
- [Optional] Override Linear Check