fix: Jaeger keeps at most 50 000 traces, and retention and sampling are listed in one table

Merged
HA
15h ago
+19-4

Summary

Closes #380. #430 already removed the four admin monitoring fields and the unused Settings fields, and #381 moved the e2e settings tests off them. What's left is Jaeger: it keeps traces in memory with its built-in cap of 100 000. Prod keeps about 12 600 traces a day at roughly 2.3 KB each, so the cap needs about 260 MiB against a 256m limit, and Jaeger would be OOM-killed after about a week and lose every trace.

Changes

Jaeger gets --set=extensions.jaeger_storage.backends.some_storage.memory.max_traces=50000 on top of its built-in config: about 4 days in roughly 145 MiB. deployment.md gets one Retention and sampling table pointing at all four values: VictoriaMetrics, Loki, Jaeger and the collector's tail sampling. Small fix found while verifying #449: the OTel SDK's Transient error ... retrying line, printed without a level when a release restarts the collector, now counts as a warning, so it no longer sends an error-lines mail.

Type of Change

  • Bug fix (non-breaking change that fixes an issue)
  • New feature (non-breaking change that adds functionality)
  • Breaking change (fix or feature that would cause existing functionality to change)
  • Documentation update
  • Refactoring (no functional changes)
  • CI/CD or tooling changes

Test Plan

  • Backend unit tests pass (cd backend && uv run pytest tests/unit)
  • Frontend unit tests pass (cd frontend && npm run test)
  • Backend E2E tests pass (docker compose exec -T backend sh -c 'uv sync --group test --no-dev --frozen --no-install-project --no-build && pytest tests/e2e')
  • Frontend E2E tests pass (cd frontend && npm run test:e2e)
  • API spec and client regenerated if routes or schemas changed (./deploy.sh openapi, then ./deploy.sh types), never edited by hand
  • Manual testing performed

Validation: jaeger:2.20.0 starts with the flag and refuses a misspelled key, so the value applies. The retry line goes from error to warning through the collector, and the 164 earlier sample lines keep their levels.

Checklist

  • My code follows the project's style guidelines
  • I have run pre-commit run --all-files and all checks pass
  • I have added tests that prove my fix/feature works
  • I have updated documentation if needed
  • My changes don't introduce new warnings

Summary by cubic

Caps Jaeger's in-memory trace storage at 50 000 traces, down from the built-in 100 000, so prod Jaeger no longer gets OOM-killed after about a week and loses every trace. Consolidates the retention and sampling settings for VictoriaMetrics, Loki, Jaeger, and the collector's tail sampling into one "Retention and sampling" table in deployment.md and links it from tracing.md.

Bug Fixes

  • The OTel SDK's Transient error ... retrying line, printed without a level, now counts as a warning, so a release restart no longer triggers an error-lines mail.