PR #11012 - opsmill/infrahub

FastAPI 0.143.0 traces requests on its own, and Infrahub must not record extra spans

This PR upgrades FastAPI from 0.141.1 to 0.143.0. Operators who turn tracing on in Infrahub receive the same spans from the API server and the task workers as before the upgrade.

Jira: INBOX-364

How tracing works in Infrahub, and what FastAPI 0.142.0 changes

When an operator turns tracing on, Infrahub records a span for each step of its work and sends the spans to a trace collector. A span is a timed record of one step: what ran, when it started, how long it took and whether it failed. For example, each API request records a request span.

Up to FastAPI 0.141.1, FastAPI recorded no spans itself. The Infrahub API is traced by a separate OpenTelemetry package, opentelemetry-instrumentation-fastapi, which server.py adds to the API when it starts.
Since FastAPI 0.142.0, FastAPI also traces requests itself (fastapi/fastapi#16403). It does this in every FastAPI app that runs in a process where tracing is turned on. For each request, it records one span for the request and one span for each step inside it: resolving dependencies, running the endpoint and, when the endpoint returns data, serializing the response.

The Release Radar bot reported that the upgrade would double every request's spans in Infrahub. The effect differs for each Infrahub process:

API server: no change in spans. FastAPI does not trace a request when it finds the separate OpenTelemetry package in the app, and server.py adds that package before the API handles its first request. This PR also turns FastAPI's own telemetry off in server.py, so the API no longer depends on FastAPI finding the package. Each API request still records one request span, and FastAPI records none. The new tests in test_server.py check this, also with the separate package removed from the app.
Task workers: three new spans for each health check, which this PR drops. With --with-healthcheck, each task worker also runs a small web server from Prefect that answers GET /health (prefect/workers/server.py). Prefect creates that server as a plain FastAPI app, so Infrahub cannot add the separate package to it or turn FastAPI's tracing off on it. When tracing is on, the worker sets it up for its whole process, so on 0.143.0 every health check would record three spans: the request, resolving dependencies and running the endpoint.
Prefect server: no change. It runs in its own container. Infrahub does not turn tracing on in that container, and Prefect 3.8.6 does not either, so FastAPI records no spans there.

Changes

fastapi 0.141.1 → 0.143.0 in pyproject.toml and uv.lock. No package is added or removed.
The API turns FastAPI's built-in tracing, metrics and logs off: server.py passes telemetry={"tracing": False, "metrics": False, "logs": False} to FastAPI(...), with a comment saying why.
Task workers drop the spans of Prefect's health checks.
- A sampler is the part of the tracing setup that decides which spans to keep. A new sampler, ExcludeServerPathsSampler in backend/infrahub/trace.py, drops the request span of every GET /health.
- It also drops every span whose parent span in the same process was dropped, so the spans inside a health check are dropped too, whatever OTEL_TRACES_SAMPLER is set to.
- Every other decision still follows OTEL_TRACES_SAMPLER, so operators who set that variable keep their sampling.
- Only the task workers use this sampler.
New knowledge page dev/knowledge/backend/observability.md: how the API server and the task workers produce traces, metrics and logs, why FastAPI's built-in telemetry stays off, and how the local observability stack collects them. It is linked from backend/AGENTS.md and dev/README.md.
New tests:
- backend/tests/unit/test_server.py checks that one API request records one request span from the separate OpenTelemetry package and no span from FastAPI's own tracing, and that FastAPI records no span even when the separate package is removed from the app.
- backend/tests/unit/test_trace.py checks that a /health request records no spans and that other requests record all their spans, with the default sampler, always_on and traceidratio.

No changelog fragment: traces contain the same spans as on 0.141.1.

How this was verified

The API test passes on both 0.141.1 and 0.143.0. It fails when the instrument_app call in server.py is removed.
The test with the separate package removed from the app fails when the telemetry setting is removed from server.py.
The sampler tests fail when the sampler is changed to never drop. The always_on and traceidratio cases also fail without the check on the parent span.
The 0.143.0 breaking change does not apply (fastapi/fastapi#16476). Since 0.143.0, FastAPI sets up exporters, the part that sends spans to a collector, only when FASTAPI_OTEL_AUTO_CONFIGURE is set, and Infrahub does not set it.
Prefect 3.8.6 accepts fastapi>=0.139.0,<1.0.0.
Local checks passed on the stable-based branch: ruff, mypy, the generated-file checks and 2782 backend unit tests.

uv run pytest backend/tests/unit/test_server.py backend/tests/unit/test_trace.py -v

Spans recorded in a local stack, before and after the upgrade

The development stack was run with tracing turned on and the spans sent to Tempo, with an image built from this branch. Only two things changed between runs: the FastAPI version, switched with uv pip install inside the server and task-worker containers, and the Infrahub code, either this branch or the commit it was then based on. The runs were done before this branch moved from develop to stable, on top of develop commit 62b99b2507, and before server.py turned FastAPI's telemetry off. stable and develop pin the same FastAPI, Prefect and OpenTelemetry versions, and the branch's commits apply to stable without changes. That setting changes no span, because FastAPI already skipped its tracing in the API. Each run sent 5 GET /api/config requests and 5 GraphQL queries (BuiltinTag { count }) to the API, and 5 health checks to each of the 2 task workers. The spans were then read back from Tempo.

Run	FastAPI	Infrahub code	Spans per `GET /api/config`	Spans per GraphQL query	Spans per task-worker health check
Before the upgrade	0.141.1	base	4	19	0
Upgrade without this PR's code	0.143.0	base	4	19	3
This PR	0.143.0	this branch	4	19	0

The API recorded the same spans in all three runs, and every span came from the separate OpenTelemetry package or from Infrahub's own tracing. For GET /api/config, these are the request span and three http send spans. FastAPI recorded no span of its own in the API.
Without this PR's code, every health check on a task worker recorded three spans from FastAPI. Prefect's health check returns its response directly, so FastAPI records no serialization span for it.
```
GET /health
├── fastapi.dependencies
└── fastapi.endpoint
```
With OTEL_TRACES_SAMPLER=always_on on the task workers and FastAPI 0.143.0, this PR before the check on the parent span (commit 981a11f4a2) still recorded fastapi.dependencies and fastapi.endpoint for every health check, without their request span. With the check (commit 73555e5a48), health checks record no span. In that run, the worker's background tasks, such as "Sync Git Repositories" and "Detect failed merges", were recorded with the same spans as before the upgrade.

Follow-up, not in this PR

A longer-term option, moving the API to FastAPI's built-in tracing and removing the separate package, is described in this comment.

🤖 Generated with Claude Code