Update Grafana dashboards: admission controller sections, panel fixes

Merged
FA
3mo ago
+8870-4272

Summary

Updates the bundled Grafana dashboards: adds monitoring sections for the new admission controller middleware and its database-stress signal, and fixes every broken panel found while auditing the bundled dashboards. docker-compose-observability-standalone.yml is regenerated from the updated sources.

Infrahub Monitoring (infrahub_monitoring.json)

New "Admission Controller" section (collapsed, 12 panels) covering all infrahub_admission_* metrics:

  • Stat band: offered load, shed rate %, slot-pool saturation vs derived capacity, % of requests without X-Priority (adoption)
  • Admission outcomes (offered = admitted + shed) and shed reason (stress/codel/backstop, stacked)
  • In-flight by priority vs pool capacity (dashed line), queued waiters by priority (the leading indicator before CoDel sheds)
  • Sojourn quantiles (reuses the $percentile variable) + full sojourn heatmap (the fast path records 0s, so the distribution is bimodal under load — quantile lines average that away)
  • Shed by priority class (verifies low sheds before medium before high)
  • Per-worker sustained-load clock (the signal that escalates Retry-After)

New "DB Stress Signal (Backpressure Input)" section (collapsed, 9 panels) covering infrahub_db_reference_query_*:

  • Stress ratio stat + timeseries against the per-class shed triggers from config.py defaults (low 10x / significant-load 20x / medium 25x / high 100x), log-scaled so the triggers stay on screen while the ratio idles near 1
  • State timeline of which priority classes are past their trigger
  • Floor vs recent-window minimum (the two ratio inputs), raw reference-query latency as a cross-check
  • Reference-query sample rate vs the warm-up floor (5 samples / 20s): below it the controller reads unstressed regardless of actual DB load

Panel descriptions document the observability gotchas found while reading the code: the ratio gauge publishes with no warm-up gate while the controller substitutes 1.0 below stress_min_samples; task-worker instances export the stress gauges as flat 0 (queries filter > 0); the queries aggregate per-pid gauge series produced under PROMETHEUS_MULTIPROC_DIR.

Other changes:

  • New GraphQL/DB query rate panels next to the existing duration panels (from the histogram _count series)
  • 12 targets migrated from hardcoded [1m] to $__rate_interval (fixes gaps at wide time ranges); the Overview row's [5m] ranges left as-is for smoothing
  • All sections collapsed by default

Neo4j (neo4j_monitoring.json)

  • Database Size showed No data: queried neo4j_database_neo4j_store_size_total, renamed in Neo4j 5.x → now store_size_database. Also dropped percent-scale thresholds (70/85 absolute) carried onto a bytes field, which would have painted the stat red the moment data appeared
  • Query Execution Time: both targets referenced _sum/_bucket series the exporter does not publish — the metric is a summary. Now reads the quantile series directly; the avg target becomes Median (p50) (no _sum exists to compute a mean) and p95 stays p95
  • Counters of Graph Items: relationship-type target referenced $job/$instance variables that don't exist on this dashboard; now uses the same hardcoded matchers as its sibling target

Prefect (prefect_flow_run_overview.json, prefect_platform_overview.json)

  • CPU/MEM usage peak errored with parse error: unexpected "=": a find/replace had eaten every literal namespace — both from the label matcher (namespace="$namespace" → ="$") and the recording-rule name (node_namespace_pod_container:... → node__pod_container:...). Restored both
  • 36 panel/target datasource refs pointed at an undefined $datasource variable (CPU/Memory Quota & Usage panels on both dashboards) → now ${datasource_prometheus}
  • Added the missing datasource_loki variable both dashboards' log panels reference

Standalone compose

docker-compose-observability-standalone.yml regenerated with development/convert_compose_standalone.py; diff is exactly the embedded dashboard content, $$ escaping intact.

Verification

  • Every new/changed PromQL expression validated against a live Prometheus fed by a running dev stack; admission and stress panels return data (shed panels are legitimately empty until a first shed occurs)
  • Dashboard JSON exported byte-compatible with each file's existing serialization, so diffs are content-only
  • docker compose config on the regenerated standalone file reports the same pre-existing task-manager dependency warning as on the base branch, nothing new
  • Not run: /pre-ci (repo venv unusable in this sandbox); nothing outside development/ JSON/YAML changed

🤖 Generated with Claude Code