feat: every container's output reaches Loki through Docker's fluentd driver and the collector

Merged
HA
1d ago
+50-10

Summary

Loki only gets what the backend and worker send over OTLP; the collector's logs pipeline had no other receiver. So Mongo, Kafka, Redis and nginx output, gunicorn's own lines and a traceback from a crash before logging starts were only visible through docker logs on the server.

Changes

Every service logs through Docker's built-in fluentd log driver to the collector's new fluent_forward receiver (127.0.0.1:24224), tagged with the container name. groupbyattrs and transform give each container its own service.name, and a filter drops the backend's and worker's JSON lines, which already arrive over OTLP with trace ids. With fluentd-async containers start while the collector is down, and docker logs reads Docker's local cache, capped at 3 × 10 MB as before. The collector stays on the local driver, or its debug exporter would echo its own lines forever. Its unused docker.sock mount goes.

Type of Change

  • Bug fix (non-breaking change that fixes an issue)
  • New feature (non-breaking change that adds functionality)
  • Breaking change (fix or feature that would cause existing functionality to change)
  • Documentation update
  • Refactoring (no functional changes)
  • CI/CD or tooling changes

Test Plan

  • Backend unit tests pass (cd backend && uv run pytest tests/unit)
  • Frontend unit tests pass (cd frontend && npm run test)
  • Backend E2E tests pass (docker compose exec -T backend sh -c 'uv sync --group test --no-dev --frozen --no-install-project --no-build && pytest tests/e2e')
  • Frontend E2E tests pass (cd frontend && npm run test:e2e)
  • API spec and client regenerated if routes or schemas changed (./deploy.sh openapi, then ./deploy.sh types), never edited by hand
  • Manual testing performed

Validation: otelcol validate passes, and so do the Grafana contract tests and mkdocs --strict. On a local observability stack, Redis lines reached Loki as service_name="redis". A container named backend got its gunicorn and traceback lines through while its JSON line was dropped. docker compose logs still works, and docker stop took 0.4 s with no collector listening.

Checklist

  • My code follows the project's style guidelines
  • I have run pre-commit run --all-files and all checks pass
  • I have added tests that prove my fix/feature works
  • I have updated documentation if needed
  • My changes don't introduce new warnings

Summary by cubic

Loki now receives the output of every container, not just the backend's and worker's OTLP lines. Mongo, Kafka, Redis, nginx, gunicorn, and pre-logging tracebacks previously only visible through docker logs now land in Grafana.

  • All compose services log through Docker's fluentd driver to the collector's new fluent_forward receiver on 127.0.0.1:24224, tagged with the container name.
  • New groupbyattrs/container and transform/container processors give each container its own service.name; filter/app_json_stdout drops the backend's and worker's JSON stdout lines, which already arrive over OTLP with trace ids.
  • fluentd-async keeps containers starting and buffering while the collector is down, and docker logs reads Docker's local cache capped at 3 × 10 MB as before.
  • The collector stays on the local driver so its debug exporter doesn't echo its own lines, and its unused docker.sock mount is removed.