feat: logs are searchable by any field in OpenSearch, next to Loki

Merged
HA
1d ago
+229-16

Summary

Closes #373. Log search in Loki is slow: on prod, one execution id over the 10 days Loki holds takes 3.4 s, and a field search with | json 8.2 s, since Loki reads every chunk. Mongo's connection and login lines are 85% of those bytes.

Changes

The collector drops Mongo's info lines and hands every log line to OpenSearch as well as Loki. structlog's JSON keys become attributes, stored as flat_object, so a key that's a number in one line and a string in another can't get a line rejected. One index a day, deleted after 30 days by a policy that start.sh applies together with the index template. Grafana gets the OpenSearch plugin and datasource. Loki's limit goes to 512m, since those searches took it to 207 MiB.

[!WARNING]
Prod gets a new container: OpenSearch 3.9 with a 512m heap, about 1.1 GiB resident at 30 days (limit 1536m), against 3.8 GB available today. Lines sent while it restarts are dropped from OpenSearch, because the exporter doesn't retry a refused connection; Loki keeps them.

Type of Change

  • Bug fix (non-breaking change that fixes an issue)
  • New feature (non-breaking change that adds functionality)
  • Breaking change (fix or feature that would cause existing functionality to change)
  • Documentation update
  • Refactoring (no functional changes)
  • CI/CD or tooling changes

Test Plan

  • Backend unit tests pass (cd backend && uv run pytest tests/unit)
  • Frontend unit tests pass (cd frontend && npm run test)
  • Backend E2E tests pass (docker compose exec -T backend sh -c 'uv sync --group test --no-dev --frozen --no-install-project --no-build && pytest tests/e2e')
  • Frontend E2E tests pass (cd frontend && npm run test:e2e)
  • API spec and client regenerated if routes or schemas changed (./deploy.sh openapi, then ./deploy.sh types), never edited by hand
  • Manual testing performed

Validation: local stack from this branch. With 1.94 M synthetic lines over 31 daily indices, a search by execution id took 6 to 8 ms in OpenSearch and 25 to 75 ms through Grafana. A first start from empty volumes applies the template and policy and installs the plugin, and a restart leaves every index green. The contract test and strict mkdocs pass.

Checklist

  • My code follows the project's style guidelines
  • I have run pre-commit run --all-files and all checks pass
  • I have added tests that prove my fix/feature works
  • I have updated documentation if needed
  • My changes don't introduce new warnings

Summary by cubic

Adds OpenSearch log search alongside Loki to address #373: searches can query indexed fields instead of scanning Loki chunks.

  • Parses structured backend and worker logs into searchable attributes, with daily indices retained for 30 days and a provisioned Grafana datasource.
  • Drops Mongo info-level logs from both stores while keeping warnings and errors, and raises Loki’s memory limit from 256 MiB to 512 MiB.
  • Adds OpenSearch to the observability stack with a 512 MiB heap and 1,536 MiB container limit; logs sent during OpenSearch startup or restarts may be missing there, but remain in Loki.
  • Documents OpenSearch setup, queries, and operational details.