fix(ci): stop OpenSearch disk-watermark flakes on test runners

Merged
RO
16d ago
+28-5

Description

Two defenses against the disk-pressure flake that killed integration-tests (tests/cli, ee) on https://github.com/onyx-dot-app/onyx/actions/runs/37497824193/job/112406116495 (23 tests errored at fixture setup: 403 index_create_block_exception / FORBIDDEN/10 → "Could not connect to a document index").

Why it happens: test runners boot on runs-on's default ~38 GB root volume (see #11271). ~50% is AMI baseline; pulling the devcontainer image, the model-server image, and the compose service set adds ~15 GB, so / hovers near 90% — exactly OpenSearch's high disk watermark, where DiskThresholdMonitor applies cluster.blocks.create_index on a single-node cluster where shards can't be relocated anyway.

Changes:

  • deployment/docker_compose/docker-compose.dev.yml (+ embedded copy): cluster.routing.allocation.disk.threshold_enabled=false — disables the watermark monitor entirely for the single-node dev/CI cluster. Also removes the flood-stage read_only_allow_delete failure mode mid-run. Prod compose untouched.
  • pr-integration-tests.yml: volume=100gb on integration-tests, onyx-lite-tests, multitenant-tests — doubles headroom (~35 GB peak usage → ~45% full), matching craft/terraform.
  • pr-python-connector-tests.yml: volume=100gb on build-backend-image, connectors-check — same class of risk (peaked 73% pulling the prod backend image).

How Has This Been Tested?

  • docker compose -f docker-compose.yml -f docker-compose.dev.yml config — cluster.routing.allocation.disk.threshold_enabled: "false" lands in the opensearch service env (translated to -E by the entrypoint, same as discovery.type / bootstrap.memory_lock).
  • Couldn't boot the container locally (Docker Hub 429); the setting is startup-validated by OpenSearch and exercised end-to-end by this PR's integration-test jobs.
  • volume=Ngb label syntax is established in-repo (#11271, craft, playwright, terraform workflows).

Additional Options

  • [Optional] Please cherry-pick this PR to the latest release version.
  • [Optional] Override Linear Check

Link to Devin session: https://app.devin.ai/sessions/139535851e8a4fe5be21da30c2360a1a
Open in Devin Desktop: https://app.devin.ai/desktop/session/139535851e8a4fe5be21da30c2360a1a?variant=devin
Requested by: @rohoswagger

Summary by cubic

Disables OpenSearch's disk watermark monitor in the dev/CI compose files and gives image-heavy test runners larger volumes so high disk usage no longer breaks index creation during test runs.

OpenSearch blocks index creation when any node crosses the 90% disk watermark and marks indexes read-only at 95%. On a single-node dev/CI cluster those watermarks only cause flake — there is no second node to relocate shards to, so a CI runner hitting 91% disk during image pulls got 403 index_create_block_exception on every index, failing all 23 tests at fixture setup via setup_onyx.

  • Adds cluster.routing.allocation.disk.threshold_enabled=false to the OpenSearch service in deployment/docker_compose/docker-compose.dev.yml and the embedded copy under cli/internal/deploy/deployfiles/embedded/docker_compose/. Dev/CI compose files only; production compose is untouched.
  • Bumps the runner volume to 100gb for the integration, onyx-lite, multitenant, and connector test jobs, since the default ~38GB volume peaked past 90% after pulling the devcontainer, model-server, and compose service images.
  • Updates the disk-pressure footnote in .agents/feature-map/components/document-index.md to note that the failure mode only applies to deployments without the dev overlay.