Update Grafana dashboards: admission controller sections, panel fixes
Summary
Updates the bundled Grafana dashboards: adds monitoring sections for the new admission controller middleware and its database-stress signal, and fixes every broken panel found while auditing the bundled dashboards. docker-compose-observability-standalone.yml is regenerated from the updated sources.
Infrahub Monitoring (infrahub_monitoring.json)
New "Admission Controller" section (collapsed, 12 panels) covering all infrahub_admission_* metrics:
- Stat band: offered load, shed rate %, slot-pool saturation vs derived capacity, % of requests without
X-Priority(adoption) - Admission outcomes (offered = admitted + shed) and shed reason (
stress/codel/backstop, stacked) - In-flight by priority vs pool capacity (dashed line), queued waiters by priority (the leading indicator before CoDel sheds)
- Sojourn quantiles (reuses the
$percentilevariable) + full sojourn heatmap (the fast path records 0s, so the distribution is bimodal under load — quantile lines average that away) - Shed by priority class (verifies low sheds before medium before high)
- Per-worker sustained-load clock (the signal that escalates
Retry-After)
New "DB Stress Signal (Backpressure Input)" section (collapsed, 9 panels) covering infrahub_db_reference_query_*:
- Stress ratio stat + timeseries against the per-class shed triggers from
config.pydefaults (low 10x / significant-load 20x / medium 25x / high 100x), log-scaled so the triggers stay on screen while the ratio idles near 1 - State timeline of which priority classes are past their trigger
- Floor vs recent-window minimum (the two ratio inputs), raw reference-query latency as a cross-check
- Reference-query sample rate vs the warm-up floor (5 samples / 20s): below it the controller reads unstressed regardless of actual DB load
Panel descriptions document the observability gotchas found while reading the code: the ratio gauge publishes with no warm-up gate while the controller substitutes 1.0 below stress_min_samples; task-worker instances export the stress gauges as flat 0 (queries filter > 0); the queries aggregate per-pid gauge series produced under PROMETHEUS_MULTIPROC_DIR.
Other changes:
- New GraphQL/DB query rate panels next to the existing duration panels (from the histogram
_countseries) - 12 targets migrated from hardcoded
[1m]to$__rate_interval(fixes gaps at wide time ranges); the Overview row's[5m]ranges left as-is for smoothing - All sections collapsed by default
Neo4j (neo4j_monitoring.json)
- Database Size showed No data: queried
neo4j_database_neo4j_store_size_total, renamed in Neo4j 5.x → nowstore_size_database. Also dropped percent-scale thresholds (70/85 absolute) carried onto a bytes field, which would have painted the stat red the moment data appeared - Query Execution Time: both targets referenced
_sum/_bucketseries the exporter does not publish — the metric is a summary. Now reads the quantile series directly; the avg target becomes Median (p50) (no_sumexists to compute a mean) and p95 stays p95 - Counters of Graph Items: relationship-type target referenced
$job/$instancevariables that don't exist on this dashboard; now uses the same hardcoded matchers as its sibling target
Prefect (prefect_flow_run_overview.json, prefect_platform_overview.json)
- CPU/MEM usage peak errored with
parse error: unexpected "=": a find/replace had eaten every literalnamespace— both from the label matcher (namespace="$namespace"→="$") and the recording-rule name (node_namespace_pod_container:...→node__pod_container:...). Restored both - 36 panel/target datasource refs pointed at an undefined
$datasourcevariable (CPU/Memory Quota & Usage panels on both dashboards) → now${datasource_prometheus} - Added the missing
datasource_lokivariable both dashboards' log panels reference
Standalone compose
docker-compose-observability-standalone.yml regenerated with development/convert_compose_standalone.py; diff is exactly the embedded dashboard content, $$ escaping intact.
Verification
- Every new/changed PromQL expression validated against a live Prometheus fed by a running dev stack; admission and stress panels return data (shed panels are legitimately empty until a first shed occurs)
- Dashboard JSON exported byte-compatible with each file's existing serialization, so diffs are content-only
docker compose configon the regenerated standalone file reports the same pre-existingtask-managerdependency warning as on the base branch, nothing new- Not run:
/pre-ci(repo venv unusable in this sandbox); nothing outsidedevelopment/JSON/YAML changed
🤖 Generated with Claude Code