refactor: the queue metrics come from spans, and QueueMetrics is gone

Merged
HA
20h ago
+270-191

Summary

Part of #389, the first area: the six queue metrics now come from spans instead of QueueMetrics. The old queue_depth and queue_active were in-process counters, so they read 0 after every worker restart while work sat in Redis.

Changes

ExecutionQueueService opens queue.enqueue and queue.schedule spans with priority, wait and outcome, and the saga scheduler's 10 s job puts the depth and active slots it reads from Redis on queue.schedule_pending. The collector's signaltometrics connector turns these into the same six series, and deltatocumulative makes them cumulative. The traces pipeline gives every span to the connector before tail sampling. Services get trace.Tracer through their constructors.

[!NOTE]
A span metric only gets a point when a matching span arrives, unlike an SDK counter that exports every 60 s. Counter rates now fall back to or vector(0), so an idle queue reads 0, and the wait-time panel groups by queue_priority.

Type of Change

  • Bug fix (non-breaking change that fixes an issue)
  • New feature (non-breaking change that adds functionality)
  • Breaking change (fix or feature that would cause existing functionality to change)
  • Documentation update
  • Refactoring (no functional changes)
  • CI/CD or tooling changes

Test Plan

  • Backend unit tests pass (cd backend && uv run pytest tests/unit)
  • Frontend unit tests pass (cd frontend && npm run test)
  • Backend E2E tests pass (docker compose exec -T backend sh -c 'uv sync --group test --no-dev --frozen --no-install-project --no-build && pytest tests/e2e')
  • Frontend E2E tests pass (cd frontend && npm run test:e2e)
  • API spec and client regenerated if routes or schemas changed (./deploy.sh openapi, then ./deploy.sh types), never edited by hand
  • Manual testing performed

Validation: unit 561 passed, contract 3 passed, e2e 488 passed and 1 skipped against a local stack whose worker sent its spans to this collector config. VictoriaMetrics then had 45 enqueued, 45 scheduled and 45 wait-time samples, and the gauges every 10 s.

Checklist

  • My code follows the project's style guidelines
  • I have run pre-commit run --all-files and all checks pass
  • I have added tests that prove my fix/feature works
  • I have updated documentation if needed
  • My changes don't introduce new warnings

Summary by cubic

Refactors the six queue metrics to come from spans instead of in-process counters, part of #389. queue_depth and queue_active previously read 0 after every worker restart while work sat in Redis; they now come from a span on the scheduler's 10 s job that reads Redis, so gauges stay accurate across restarts.

Notes

  • ExecutionQueueService opens queue.enqueue and queue.schedule spans with priority, wait time and outcome; the saga scheduler reports depth and active slots on queue.schedule_pending.
  • The collector's signaltometrics connector turns those spans into the same six series, and deltatocumulative makes them cumulative; the traces pipeline hands every span to the connector before tail sampling.
  • A span metric only gets a point when a matching span arrives, so counter rates now fall back to 0 with or vector(0), and the wait-time panel groups by queue_priority.
  • QueueMetrics is removed; services get a trace.Tracer through their constructors.