fix(indexing): do not commit a chunk count the index did not take

Merged
DA
4d ago
+123-3

Split out of #16178, which needed this fix but is otherwise about letting a caller re-process content it did not fetch. Unlike the opt-in flags there, this one changes behaviour for every caller, so it stands on its own.

The bug

post_index wrote the new chunk_count for every updatable document, including ones whose embedding or vector-db write had failed. Their old chunks are still in the index, so the row then described a state that does not exist, and any sweep keyed on an outdated chunk_count skipped them.

The failed ids now reach post_index, and the count is written only for the rest — the same rule the content hash beside it already followed.

Both failure classes carry a document id, so the exclusion catches both: embedding failures are built per document by _document_failure (embedder.py), and vector-db write failures per document in vector_db_insertion.py. _verify_indexing_completeness raises if an updatable document is neither inserted nor reported failed, so there is no third category to miss.

The same bug in the user file adapter

Found while reviewing this PR. UserFileIndexingAdapter.post_index marked every updatable file COMPLETED, wrote its chunk count and token count, stored its plaintext, and ran _notify_assistant_owners_if_files_ready.

_process_user_file_with_indexing does set FAILED afterwards, in its own session, so the status recovers. The notification does not: the owner is told "All files for agent X have been processed and are now available" for a file that then shows as failed. It fires whenever the failed file was the last one that agent was waiting on.

The adapter now skips the failed ids, leaving the status for the caller to fail.

Testing

test_post_index_leaves_chunk_count_alone_for_a_failed_write covers the connector path; test_post_index_skips_a_failed_user_file covers the user file path, and fails without the fix. 218 unit tests pass in backend/tests/unit/onyx/indexing/. In test_persona_file_sync.py the same 13 tests fail before and after the change — they need MinIO and OpenSearch, which this environment does not run.

🤖 Generated with Claude Code

Summary by CodeRabbit

  • Bug Fixes

    • Files that fail during indexing no longer receive updated chunk counts or completion metadata, trigger assistant-ready notifications, or have plaintext stored.
    • Successful files continue to receive indexing updates as expected.

  • Tests

    • Added coverage to verify failed files remain in progress and are excluded from chunk-count updates.