fix: a release restarts the observability services whose mounted config changed

Merged
HA
10d ago
+15-1

Summary

Found while working on #317. Release 2026.10.28 shipped new span rules in otel-collector-config.yaml, but the collector kept its old config for 2.5 hours, so two executions never got their metrics. The deploy checks out the release tag and runs docker compose up -d, and compose recreates a container only when its service definition changes (hash.go), never for a new file behind a mount. The collector only picked up the rules when #409 bumped its image the next night.

Changes

The deploy remembers the commit the server ran before the checkout. After up, it restarts the collector, VictoriaMetrics, Loki or Grafana when the release changed a config that service mounts. Grafana re-reads its dashboards every 10 seconds, so a dashboard-only change doesn't restart it. A running container can't reload instead: git writes a new inode, and a file bind mount keeps the old one.

Type of Change

  • Bug fix (non-breaking change that fixes an issue)
  • New feature (non-breaking change that adds functionality)
  • Breaking change (fix or feature that would cause existing functionality to change)
  • Documentation update
  • Refactoring (no functional changes)
  • CI/CD or tooling changes

Test Plan

  • Backend unit tests pass (cd backend && uv run pytest tests/unit)
  • Frontend unit tests pass (cd frontend && npm run test)
  • Backend E2E tests pass (docker compose exec -T backend sh -c 'uv sync --group test --no-dev --frozen --no-install-project --no-build && pytest tests/e2e')
  • Frontend E2E tests pass (cd frontend && npm run test:e2e)
  • API spec and client regenerated if routes or schemas changed (./deploy.sh openapi, then ./deploy.sh types), never edited by hand
  • Manual testing performed

Validation: the restart lines over the last four releases pick the collector and VictoriaMetrics for 2026.10.28, only the collector for 2026.10.29 and nothing for the two after, matching what changed. A local container saw a replaced mounted file only after docker restart. actionlint reports nothing new; the script runs only on the next release.

Checklist

  • My code follows the project's style guidelines
  • I have run pre-commit run --all-files and all checks pass
  • I have added tests that prove my fix/feature works
  • I have updated documentation if needed
  • My changes don't introduce new warnings

Summary by cubic

Fixes the deploy so a release restarts observability services whose mounted config changed; previously the collector ran 2.5 hours on its old config after release 2026.10.28 shipped new span rules, so two executions lost their metrics.

  • Records the commit the server ran before the checkout and diffs it against the release tag for each mounted config path.
  • Restarts otel-collector, victoria-metrics, loki or grafana when their config changed, because docker compose up recreates a container only when its service definition changes and a bind mount keeps the old inode.
  • Skips Grafana for dashboard-only changes since it re-reads dashboards every 10 seconds.
  • Documents the restart behavior and the matching manual rollback step.