fix: VictoriaMetrics scrapes span counters, so rare events count right

Merged
HA
19h ago
+46-5

Summary

Part of #389. Since #392 the queue counters come from spans, and the collector's signaltometrics connector sends a point only when a span arrives. Once two points are more than about 8 to 10 minutes apart, VictoriaMetrics loses the earlier one, so increase counts the whole total again and rate returns nothing (measured on v1.153.0).

Changes

Span counters and histograms now go through deltatocumulative to the collector's prometheus exporter, and VictoriaMetrics scrapes it every 15 s, so each series gets a point per scrape. max_stale: 24h keeps an idle counter's total. Span gauges stay on remote write, split off by two filter processors, so a gauge still goes stale when its job stops. Names and labels don't change.

Type of Change

  • Bug fix (non-breaking change that fixes an issue)
  • New feature (non-breaking change that adds functionality)
  • Breaking change (fix or feature that would cause existing functionality to change)
  • Documentation update
  • Refactoring (no functional changes)
  • CI/CD or tooling changes

Test Plan

  • Backend unit tests pass (cd backend && uv run pytest tests/unit)
  • Frontend unit tests pass (cd frontend && npm run test)
  • Backend E2E tests pass (docker compose exec -T backend sh -c 'uv sync --group test --no-dev --frozen --no-install-project --no-build && pytest tests/e2e')
  • Frontend E2E tests pass (cd frontend && npm run test:e2e)
  • API spec and client regenerated if routes or schemas changed (./deploy.sh openapi, then ./deploy.sh types), never edited by hand
  • Manual testing performed

Validation: e2e 488 passed and 1 skipped on a local stack sending its spans to this collector. The queue counts rose to 45 and held there, one point per scrape. Synthetic spans in bursts of 3, 2 and 4 read 9; 2 spans, 400 s of quiet, then 1 more read 3.

Checklist

  • My code follows the project's style guidelines
  • I have run pre-commit run --all-files and all checks pass
  • I have added tests that prove my fix/feature works
  • I have updated documentation if needed
  • My changes don't introduce new warnings

Summary by cubic

Fixes span counter metrics so rare events are counted correctly.

Previously, signaltometrics sent a point only when a span arrived, so over long idle gaps VictoriaMetrics dropped the earlier point, causing increase to count the total again and rate to return nothing. Span counters and histograms now go through deltatocumulative to the collector's prometheus exporter, which VictoriaMetrics scrapes every 15 seconds. Idle counters keep their total for 24 hours. Span gauges stay on remote write so they still go stale when their job stops.