feat(indexing): support callers that re-process already-indexed content
Shared indexing write-path changes, split out of #16153 (tabular re-chunk migration), which stacks on this and is the first caller.
A failed write committed a chunk count the index never took
The only change here that is not opt-in, and the one worth reviewing first.
post_index wrote the new chunk_count for every updatable document, including ones whose embedding or vector-db write had failed. Their old chunks are still in the index, so the row described a state that does not exist, and any sweep keyed on an outdated chunk_count skipped them. The failed ids now reach post_index and the count is written only for the rest, as the content hash beside it already was.
This applies to every caller, connector indexing included.
UserFileIndexingAdapter.post_index ignores the new argument: one user file is one batch, and _process_user_file_with_indexing fails the whole file when the pipeline reports any failure at all.
Ancestors were only resolvable from the connector's raw id
_get_ancestor_ids_for_documents resolved exclusively from parent_hierarchy_raw_node_id. A document reconstructed from its stored row carries the resolved parent_hierarchy_node_id but not the raw one, so its ancestors collapsed to the SOURCE node and it dropped out of folder-scoped search. Nothing hits this on main today — #16153 is the first caller that rebuilds a document from its row.
The adapter now falls back to the resolved id when, and only when, the document has no raw id. The raw id keeps priority because resolving it refreshes the hierarchy cache on a miss: docfetching resolves parent_hierarchy_node_id for every connector document and falls back to the SOURCE node without refreshing, so that value is indistinguishable from a document that really sits at the root. Preferring it would have walked from SOURCE for any document whose parent was not cached at docfetch time.
get_ancestors_from_node_id is the entry point _walk_ancestor_chain's own docstring already described — "Internal helper used by both get_ancestors_from_node_id and get_ancestors_from_raw_id" — but which nothing provided.
Two opt-in flags on run_indexing_pipeline
Both default to false. A connector path must not pass either.
force_update skips both change gates. It existed only on the per-document retry inside index_doc_batch_with_handler, so a caller with no fresher content had no way past them. For a tabular document there is no way at all: content_hash() walks Text and Image sections only, so a workbook's hash never moves however much its contents change.
skip_document_row_writes leaves the prepare-time upsert, its staged-file promotion and its replaced-blob deletes alone. post_index still writes the row's timestamps and chunk count. A caller re-processing content it didn't fetch has nothing newer to write, and reproducing every row value just so the upsert can write it back unchanged is a standing invitation to drop one — which is where both preservation bugs found while reviewing #16153 lived. Chunk enrichment still reads the live rows, so chunks stay correct.
Verified in #16153 by diffing all fifteen Document columns across a re-chunk: only chunk_count moved.
Testing
4,482 unit tests pass across indexing, db and connectors. One failure in test_retry_after_caps.py reproduces identically on a clean tree with the same suite combination — pre-existing test-ordering pollution.
🤖 Generated with Claude Code