PR #15336 - onyx-dot-app/onyx

Description

Onyx no longer uses Vespa. Every instance on the current version runs on OpenSearch (ONYX_DISABLE_VESPA already defaulted to true). This PR removes the dead Vespa code and the Vespa to OpenSearch migration, and consolidates the interfaces that only existed so that both backends could run at the same time. Instances on the latest version see no behavior change.

Removed

Vespa document index: onyx/document_index/vespa/, vespa_constants.py, the app config and schema templates, and the EE Vespa Cloud config.
Vespa to OpenSearch migration: the Celery task, beat entry, opensearch_migration queue, Redis lock, task names, the /admin/opensearch-migration API, DB helpers, and configs.
Migration tracking tables: opensearch_document_migration_record and opensearch_tenant_migration_record, plus their ORM models and enums. The new migration 3067343245d1 drops them; its downgrade recreates them.
Code that only acted on Vespa errors:
- the HttpxPool Vespa client,
- the RetryDocumentIndex wrapper (it only retried httpx.ReadTimeout, which OpenSearch never raises),
- the httpx.HTTPStatusError / tenacity branches in the Celery exception handlers,
- the 507 Vespa log hint.
Dead knowledge graph (KG) extraction pipeline: Celery stopped running KG extraction and clustering in #7908; after that only unit tests imported them. Removed:
- onyx/kg/extractions/, onyx/kg/clustering/, reset_source.py, and the extraction, embedding, lock, and formatting utils;
- the KG DB helpers, models, prompts, and configs that only the pipeline used, plus the unused process_kg_commands;
- the KG_DOCUMENT_CLASSIFICATION / KG_DEEP_EXTRACTION LLM flows and their admin usage labels. The LLMFlow registry came after the pipeline stopped, so no usage records use these flows;
- the KG unit tests for the pipeline (test_kg_extraction_tracing.py, test_kg_document_chunk_batches.py).
- The KG admin API (/admin/kg), the KG tool, and the KG cleanup in document deletion stay.
Configs and env vars:
- ONYX_DISABLE_VESPA, ENABLE_OPENSEARCH_INDEXING_FOR_ONYX, ENABLE_OPENSEARCH_RETRIEVAL_FOR_ONYX, DISABLE_OPENSEARCH_MIGRATION_TASK
- VESPA_*, MANAGED_VESPA
- RECENCY_BIAS_MULTIPLIER, RERANK_COUNT, LOG_VESPA_TIMING_INFORMATION, VESPA_LANGUAGE_OVERRIDE, VESPA_SEARCHER_THREADS
- The KG configs that only the removed pipeline read (KG_CLUSTERING_*, KG_NORMALIZATION_*, KG_SQL_GENERATION_*, and similar).
- The env-var drift baseline drops exactly these entries. Deployments that still set them are not affected, because nothing reads them.
Scripts and tests: Vespa-only dev scripts and tests.
Deployment:
- Helm: the legacyVespaCheck guard and two dead env vars. Chart bumped to 0.9.1.
- Compose: VESPA_SEARCHER_THREADS; the files were regenerated with ods generate-compose.
- CloudFormation: the Vespa service.
- CI: the Vespa Helm repos.
- Devcontainer, .vscode, .zed, docs, and the zip package in the backend image (only Vespa used it).

Consolidated

One factory function: get_all_document_indices (a list, for Vespa + OpenSearch dual writes) is merged into get_default_document_index, which returns one DocumentIndex (the OpenSearch primary/secondary pair). The indexing pipeline, Celery tasks, setup, index swap, and ingestion take one index instead of a list. primary_backfill_in_progress is now keyword-only, so a stale positional call fails loudly.
One interface module: onyx/document_index/interfaces_new.py is renamed to interfaces.py. The old interface module was already gone.
Test client: the integration helper vespa_fixture is now DocumentIndexClient. It returns OpenSearch chunks directly instead of reshaping them into the Vespa format.
Old migrations: 12635f6655b7 and 90e3b9af7da4 imported Vespa helpers, so those helpers are now inlined in the two migrations. The inlined HTTP client is the same as the removed helper: managed Vespa (MANAGED_VESPA=true) uses the VESPA_CLOUD_CERT_PATH / VESPA_CLOUD_KEY_PATH client certificate and verifies TLS, self-hosted Vespa uses verify=False, and VESPA_REQUEST_TIMEOUT sets the timeout. No other old migration changed.

Kept on purpose, so running deployments are not disrupted

Live Celery names that still say "vespa": the vespa_metadata_sync queue, check_for_vespa_sync_task, the tasks/vespa/ module, the beat lock, and the Prometheus labels. They sync OpenSearch. Renaming them would strand queued tasks and break dashboards.
opensearch_indexing_enabled in the settings API: now always true, because the published Terraform provider exposes it. The provider description marks it as deprecated.
Terraform node group names and vespa_node_* variables: renaming them would replace live node groups. Only their descriptions changed.
CloudFormation Vespa EFS access points: old stacks that import them can still update.

Behavior changes

Search expansion fix: _retrieve_adjacent_chunks no longer rewrites ' to _ in document IDs. OpenSearch stores raw IDs, so on main the chunks around a hit silently came back empty for any document whose ID has an apostrophe. A new test fails on main and passes here.
/versions: it now reports opensearchproject/opensearch:3.6.0 as the index image.
Table-drop migration: dropping the table with the foreign key to document needs an exclusive lock on document. The migration uses a 5 s lock_timeout inside a savepoint and retries, so a long indexing transaction causes short stalls instead of one long queue. It retries only a lock timeout (SQLSTATE 55P03); any other error fails the migration at once.
Rolling deploy: old pods serving /admin/opensearch-migration return 500 once the tables are gone. Nothing in the UI calls that endpoint.
scripts/orphan_doc_cleanup_script.py:
- It deletes orphan chunks from the primary and the secondary index, so an index swap does not leave chunks in the future index.
- It no longer looks up each document before the delete. That lookup read only the primary, so it skipped documents that existed only in the future index.
- It stops when no document in a batch can be deleted. Before, it retried the same batch forever.
scripts/restart_containers.sh: it still removes the onyx_vespa container that older versions started. It rejects the old four-argument form, so a Vespa volume is never mounted as the Postgres data volume.
Lite (DISABLE_VECTOR_DB): the vector-quantization check now skips on these deployments instead of trying to reach a missing OpenSearch.

How Has This Been Tested?

New automated tests (run in CI)

Area	Tests
Unit	`test_document_index_factory.py`: disabled vector DB, primary only, primary+secondary, backfill flag, keyword-only flag. `test_setup_document_index.py`: retries, attempt limit, write-blocked index. `test_search_utils.py`: `_retrieve_adjacent_chunks` with raw IDs, ranges, sorting, errors. `test_settings_store.py`: `opensearch_indexing_enabled` always true. `test_get_versions.py`: `/versions` index image and its match with the compose file.
External dependency (real OpenSearch)	`test_index_pair_search_and_permissions.py`: pair routing (writes and reads use the primary only); access, document-set, and hidden updates reach both indices and retrieval enforces them (id-based and keyword); delete fan-out; section expansion for a plain ID and for an ID with `'`.
External dependency (Postgres + Celery eager)	Retry exhaustion: the metadata sync task fails after `max_retries` and leaves the doc stale; the cc_pair cleanup task hands the doc to reconciliation on its last retry.
External dependency (real Postgres + OpenSearch)	`scripts/test_orphan_doc_cleanup_script.py`: orphans in both indices, only in the secondary, and in neither are deleted from both indices and from Postgres; when every delete fails, the script stops after one batch and keeps the documents.
Migration	`test_drop_opensearch_migration_tables.py`: drops populated tables and keeps documents; downgrade recreates the tables (FK, defaults, singleton index); waits out a 7 s lock on `document`; fails at once on an error that is not a lock timeout (a dependent view).

One-off local verification

The integration and external-dependency runs used an isolated Compose project on alternate ports: Postgres, Redis, OpenSearch, and the SeaweedFS object store. The in-process API server and the Celery workers ran from this branch. Scratch databases, buckets, containers, and test indexes were removed afterwards.

Static checks

prek 0.3.4 over origin/main..HEAD passes every hook: ruff, ruff format, ty, getattr and lazy-import checks, compose sync, env-drift baseline, TypeScript type check and coverage, oxlint, oxfmt, mobile type check / lint / format, terraform fmt and validate, zizmor, shellcheck, check-yaml, and ripsecrets.
golangci-lint and actionlint could not install locally (a TLS proxy blocks the Go download). I ran actionlint 1.7.9 in Docker on the four changed workflows (clean) and go vet in Docker (below).
Full ty check and ruff check on backend/ pass.

Unit tests

pytest backend/tests/unit: 11,207 passed, 32 skipped (after the KG pipeline tests were deleted; 11,212 before).
The license tests only pass here with Redis pointed at an unused port: they use a real Redis throttle key, which collides with a local Redis. That happens on main too.

External-dependency tests

The search and permission suites (opensearch/, document_index/, hierarchy/, celery/, indexing/, db/test_port_metadata_defer_e2e.py): 271 passed.
Baselines run on an origin/main export:
- The apostrophe expansion test fails on main ([] == [0, 1]) and passes here.
- The new metadata-sync retry test passes on both, so retry behavior is unchanged.
- The changed deletion, swap, and search-settings suites have the same environmental failures on both sides (S3 credentials, an index-name collision with the local dev index). There are no new failures.

Integration tests (host pytest against the isolated stack; OPENAI_API_KEY for LLM calls)

Suite	Result
`tests/search` (document-set filtering, ACL enforcement, persona scope, PAT scope)	7/7
`tests/search_settings`	10/10
`tests/document_set` (sync to the index)	3/3
`tests/usergroup`	36/36
`tests/permissions`	61 passed, 1 xfail
`tests/permissions_access`	400/400
`tests/permissions_membership`	248/248
`tests/permissions_resources`	216/216
`tests/connector` (deletion verifies ACLs and document sets in OpenSearch)	9/11 (2 need Confluence credentials)
`tests/indexing` (with the mock connector server)	11/11
`tests/ingestion`	1/1
`tests/reindex_port` (primary + secondary pair end to end)	7/7
`tests/chat`	19/19
`tests/streaming_endpoints`	16/16
`tests/image_indexing`	2/2
`tests/projects`, `tests/tags`, `tests/index_attempt`, `tests/personas`	1/1, 2/2, 1/1, 23/23
`tests/no_vectordb` (Lite: `DISABLE_VECTOR_DB=true`, Postgres-only)	18/18

About the local runs:

Reruns: the first runs of search, usergroup, and permissions_access failed only because the LLM key or TLS setup was missing. Once configured, the affected tests were re-run and passed.
search_settings: it was re-run after another workload on the Docker host caused an OOM kill of the isolated OpenSearch.
streaming_endpoints: 1 of 16 first failed on ranking, because the local runs share one OpenSearch across suites. It passed 16/16 on a clean index. CI uses a fresh stack per suite.
tests/pruning: it can't run on macOS, because Playwright installs a Linux browser build. CI covers it.

Migrations

pytest tests/integration/tests/migrations/ on a throwaway Postgres: 18 passed, 1 skipped. That includes pytest-alembic test_upgrade, test_single_head_revision, and test_up_down_consistency over every revision, including the new migration and the two edited ones.
Manual checks on scratch databases:
- The full chain runs to head.
- The upgrade drops populated tables and leaves document rows alone.
- After a downgrade, the columns, indexes, and constraints match what the original migrations build, exactly.
- With a separate session holding a lock on document for 18 s, the drop retried twice and then succeeded. Concurrent reads of document stalled at most 5 s.
alembic heads / history load every revision for both alembic and schema_private.

Helm

helm dependency build and helm lint pass.
helm template renders with values.yaml, values-ci.yaml, and values-lite.yaml.
A rendered diff against origin/main (ignoring the chart version label and config checksum) shows only the removed env vars and the removed opensearch_migration queue.

Terraform provider: gofmt, go vet, go build, and go test ./... pass (Docker, Go 1.27.1).
The provider docs were edited to match the new schema description. tfplugindocs could not download locally, and CI's docs-drift check covers it.
CLI: go vet passes. go test ./... passes except TestRerunAsksOnceAndCancels, which fails the same way on origin/main. The embedded-files sync test (deployfiles) passes.

KG removal

find_dead.py (an AST script) listed top-level symbols that nothing outside their own file references. It ran on the KG modules and the KG DB, config, and prompt files until no new dead symbols appeared. Only symbols that the removed pipeline used were deleted. Unrelated symbols that were already dead on main stay.
Every module under onyx and ee (857) imports without errors. ty check and ruff check on backend/ pass.
tests/integration/tests/kg (the KG admin API, including reset): 4/4.
tests/integration/tests/connector: both deletion tests pass. They run the KG cleanup that document deletion still calls. The 2 Confluence tests need credentials, as before.
External dependency: indexing/test_document_deletion_file_cleanup.py, celery/test_document_sync_no_pg_txn_during_index_io.py, and document_index/: 40/40. The pool-checkout test first failed only because the run set POSTGRES_USE_NULL_POOL=true; it passed 4/4 on a normal pool.

Inlined Vespa client in the old migrations

Loaded each of the two migration modules and captured the httpx.Client arguments. The script also built a real client each time:
- self-hosted: no client certificate, verify=False, timeout 15, endpoint http://localhost:8081;
- managed, with a generated certificate pair: client certificate set, verify=True, timeout from VESPA_REQUEST_TIMEOUT (30), endpoint from VESPA_CLOUD_URL;
- managed, with a missing certificate file: FileNotFoundError, the same as the removed helper.

Second review round (cubic)

Each new test was also run against the code before its fix, to show that it catches the problem:
- Drop migration, dependent view: the pre-fix migration took 55.4 s to fail. The fixed one fails in under 5 s.
- Orphan cleanup, pre-fix script: 3 chunks of the document in both indices stay in the secondary.
- Orphan cleanup, with only the secondary passed in: 3 chunks of the secondary-only document stay in the secondary. That is why the lookup was also removed.
- Orphan cleanup, without the stop guard: the test fails with "the script keeps retrying the same batch".
tests/integration/tests/migrations/: 19 passed, 1 skipped (one new test).
Web: jest for SystemUsagePanel and the i18n catalog checks, 28/28.
Pre-commit hooks on the changed files pass: ruff, ty, the getattr check, and shellcheck.

Other

/versions API test against live DockerHub: passed.
The release-process docs showed Vespa in the /versions example. onyx-dot-app/documentation#499 changes it to OpenSearch.
ods generate-compose --write regenerated the compose files and the CLI embedded copies.

CI notes

connector-filter-eval is an LLM evaluation with continue-on-error. It is information only. The same 2 of 27 cases (connector-as-topic-not-directive, policy-about-source-tool) fail on other branches, for example a Dependabot branch and nikg/object-store-service. This PR does not change the filter-extraction code.
audit fails on the advisory GHSA-jhr6-gm9c-rqjv (sentence-transformers). It fails on every recent PR, and this PR changes no dependencies.
The visual regression report shows small screenshot changes. The usage-page change is the date range in the description, which changes every day. The other changed pages are not touched by this PR.
The ECS CloudFormation templates had no OpenSearch service before this PR. This PR removes only the unused Vespa service from them.

Additional Options

[Optional] Please cherry-pick this PR to the latest release version.
[Optional] Override Linear Check

🤖 Generated with Claude Code

Summary by cubic

Removes the Vespa document index and the Vespa-to-OpenSearch migration, making OpenSearch the only document index backend, and removes the dead knowledge graph extraction pipeline. Instances already on OpenSearch see no migration; the migration tracking tables are dropped by a new table-drop migration.

Key changes

The document index factory now returns a single DocumentIndex instead of a list for dual writes, and interfaces_new is renamed to interfaces.
Removed the entire Vespa backend: document index, app configs, request builders, migration Celery task, queue, admin API, DB helpers, configs, scripts, tests, and deployment entries (Helm, Compose, CloudFormation, CI, devcontainer, docs).
Removed the HttpxPool client, RetryDocumentIndex wrapper, and error handling that only acted on Vespa errors.
Removed the dead knowledge graph extraction pipeline (onyx/kg/extractions, clustering, and the reset, embedding, and extraction utils, plus the KG DB helpers, models, prompts, and configs only it used), including the KG_DOCUMENT_CLASSIFICATION and KG_DEEP_EXTRACTION LLM flows. The KG admin API, KG tool, and KG cleanup in document deletion stay.
Vespa-named Celery artifacts are kept: vespa_metadata_sync, check_for_vespa_sync_task, and tasks/vespa/ still sync OpenSearch so queued tasks and dashboards are not disrupted. Two old migrations still call Vespa and now inline the HTTP client, honoring managed-Vespa client certificate auth.
The deprecated opensearch_indexing_enabled settings field now always returns true for the Terraform provider; Terraform node group names and CloudFormation EFS access points are left untouched so live stacks are not replaced.

Behavior changes

Search expansion no longer rewrites ' to _ in document IDs, so chunks around a hit come back correctly for IDs with apostrophes.
/versions now reports the OpenSearch image instead of the Vespa image.
Dropping the migration tables waits on the document lock with a bounded timeout instead of stalling all queries indefinitely. The drop retries only on a lock timeout; other errors fail immediately.
Orphan cleanup in orphan_doc_cleanup_script.py now builds the index from both active search settings, deletes from the future index without a primary-only lookup, and stops when a batch has no deletable documents instead of looping forever. restart_containers.sh rejects its old four-argument form.