refactor: OpenTelemetry instrumentations record HTTP and process metrics

Merged
HA
6d ago
+130-377

The FastAPI OpenTelemetry instrumentation never worked. dishka_lifespan.py called instrument_app inside the lifespan, but Starlette builds the middleware stack on its first ASGI call, which is the lifespan startup itself, and instrument_app only patches build_middleware_stack without rebuilding. So no request produced an OTel span or HTTP metric, and MetricsMiddleware was the only HTTP telemetry. A probe that drives the lifespan the way uvicorn does recorded nothing with the old placement and all three HTTP instruments with the new one.

create_app now instruments the app before the stack exists, so the OTel middleware wraps everything, and MetricsMiddleware is deleted. The backend image sets OTEL_SEMCONV_STABILITY_OPT_IN=http (pytest's env does too), so requests land as http_server_request_duration_seconds, http_server_active_requests and the two body size histograms, labelled by FastAPI's route template instead of our regexes. Since the OTel middleware is outermost, the 403, 429 and 413 answers from CSRF, rate limiting and the size limit now show up in the metrics; before they didn't.

The psutil gauges go as well. The collector's host_metrics receiver already scraped host memory, and now host CPU utilization too, and every backend and worker process reports its own memory, CPU, threads and file descriptors through opentelemetry-instrumentation-system-metrics. setup_metrics moves to app/core/metrics/base.py, and psutil is no longer a direct dependency.

The HTTP dashboard's panels read the new series. The contract test builds the instrumentations' families on a bare app the way create_app configures them, and passes. The middleware, metrics reference and contract testing docs describe the new setup.

Checked with ruff, mypy, grimp, 564 unit tests, the 3 contract tests and a strict docs build. Against the real create_app with an in-memory reader, a request recorded http.server.request.duration with method, status and scheme, and the security headers still came back.

Summary by cubic

Fixes the FastAPI OpenTelemetry instrumentation, which ran inside the lifespan after Starlette had already built the middleware stack and so never wrapped a single request. HTTP and process metrics now come from OpenTelemetry instrumentations, and the hand-written MetricsMiddleware and psutil gauges are removed.

HTTP metrics

  • Requests record http_server_request_duration_seconds, http_server_active_requests and the two body size histograms, labelled by FastAPI route templates instead of regex-normalized paths.
  • The stable semantic convention names require OTEL_SEMCONV_STABILITY_OPT_IN=http, now set in the backend image and pytest.
  • The OTel middleware wraps CSRF, rate limiting and the size limit, so their 403, 429 and 413 responses show up in metrics where they were absent before.

Process metrics

  • Each backend and worker process reports its own memory, CPU, threads and file descriptors via opentelemetry-instrumentation-system-metrics; host CPU and memory come from the collector's host_metrics receiver.
  • setup_metrics moved to app/core/metrics/base.py, and psutil stops being a direct dependency.

The Grafana HTTP dashboard, the metrics reference and middleware docs, and the contract test now match the new series.