feat(indexing): support callers that re-process already-indexed content (#16178) to release v4.8
Cherry-pick of #16178 to release v4.8.
Upstream commit: c6aded8ebe4da927a27932e7086561f939049706
Conflict resolution. Four hunks in indexing_pipeline.py conflicted because upstream's context carries skip_spreadsheet_content, a parameter that does not exist on this branch. Each hunk takes the upstream side minus that parameter: force_update and skip_document_row_writes are added to index_doc_batch_with_handler, index_doc_batch_prepare, index_doc_batch and run_indexing_pipeline, and the skip_spreadsheet_content paragraph is dropped from the run_indexing_pipeline docstring. The adapters, models.py and redis_hierarchy.py applied unchanged.
Verified: tests/unit/onyx/indexing passes (211 passed); ruff check clean.
Stacked backport. Merge in order: #16296 → #16298 → #16308 (#15563, which this needs for force_update) → this PR → the #16153 backport.
Summary by cubic
Cherry-picks #16178 to the v4.8 release branch. Upstream's skip_spreadsheet_content parameter doesn't exist on this branch, so the conflicted hunks in indexing_pipeline.py apply the upstream changes minus that parameter; the adapters, models.py and redis_hierarchy.py applied unchanged.
The pipeline now supports callers that re-process already-indexed content. Two opt-in flags on index_doc_batch and run_indexing_pipeline: force_update bypasses both change gates, and skip_document_row_writes leaves the prepare-time DB upsert, staged-file promotion and replaced-blob deletes alone while post-index timestamp and chunk-count writes still happen. Both default to false; a connector path must pass neither.
Two non-opt-in fixes ride along: post_index now writes chunk_count only for documents whose embedding or vector-db write succeeded, and document ancestor lookup falls back to parent_hierarchy_node_id when a document has no raw id (get_ancestors_from_node_id replaces the private _walk_ancestor_chain helper). UserFileIndexingAdapter ignores the new flag because one user file is one batch and a failure fails the whole file.
Stacked backport — merge in order: #16296 → #16298 → #16308 → this PR → the #16153 backport.
- [Optional] Override Linear Check