feat: every container's errors in one Errors dashboard, with alerts that reach someone

Merged
HA
2d ago
+860-36

Summary

Part of #365. Every container's lines reach Loki since #421, but nothing reads them, and the only alerts are two host-memory rules. A full disk, a crash-looping container, a burst of 5xx or a growing DLQ would reach nobody, and prod telemetry calls itself development.

[!WARNING]
The deploy now stops when ALERT_HEARTBEAT_URL or a Mailjet secret is empty. Set up the outside monitor and add that secret before this merges, or the next release fails.

Changes

The collector gives every Docker line a level from its own format before Loki guesses wrong, and reads every container through a read-only docker-socket-proxy. A new Errors dashboard shows error and warning lines per service, restarts and uptime. New mailed rules cover disk, containers, error lines, 5xx, Kafka lag, MongoDB and the DLQ, plus a heartbeat to an outside monitor. Three fixes keep the log alerts quiet: the pod watch runs as one task (no skip warning every 5 seconds, no traceback per deploy), each OTLP log record goes out once instead of twice, and /favicon.ico serves favicon.png. ENVIRONMENT is read like the secrets; the deploy sets production. Kubernetes events and browser errors aren't here.

Type of Change

  • Bug fix (non-breaking change that fixes an issue)
  • New feature (non-breaking change that adds functionality)
  • Breaking change (fix or feature that would cause existing functionality to change)
  • Documentation update
  • Refactoring (no functional changes)
  • CI/CD or tooling changes

Test Plan

  • Backend unit tests pass (cd backend && uv run pytest tests/unit)
  • Frontend unit tests pass (cd frontend && npm run test)
  • Backend E2E tests pass (docker compose exec -T backend sh -c 'uv sync --group test --no-dev --frozen --no-install-project --no-build && pytest tests/e2e')
  • Frontend E2E tests pass (cd frontend && npm run test:e2e)
  • API spec and client regenerated if routes or schemas changed (./deploy.sh openapi, then ./deploy.sh types), never edited by hand
  • Manual testing performed

Validation: 164 prod log lines sent through Docker's fluentd driver got the right levels, the proxy refused writes with 403, all 13 rule queries ran on prod, and Grafana 13.2.4 provisioned everything with 0 errors.

Checklist

  • My code follows the project's style guidelines
  • I have run pre-commit run --all-files and all checks pass
  • I have added tests that prove my fix/feature works
  • I have updated documentation if needed
  • My changes don't introduce new warnings