Why

The activity log intermittently stalls for 30s+ — the page sits on "Loading..." while every other request on the instance answers in milliseconds. In e2e CI this surfaced as the test_filter_by_account flake (runs 31707450288, 32490367256): the GET_INFRAHUB_EVENTS request black-holes, the filter bar never renders, and the test dies on a 30s click timeout while sibling GraphQL queries fired the same millisecond complete in ~30ms.

Root cause, reproduced on a CI-parity testcontainers stack: the task manager's asyncpg connections execute the /infrahub/events/filter count/read statements as named prepared statements, and after five executions Postgres switches them to a generic plan chosen without seeing the parameter values. For these filters (wide occurred window + JSON label match on event_resources) the generic plan is a quadratic nested-loop semi join. Measured through the live endpoint: executions 1–5 take ~20ms, execution 6 onward takes 8.9s at just 5k events / 40k resources, growing quadratically with event volume. The flip is per pool connection — one connection runs the pathological plan while its siblings stay fast — which is why it presented as a rare flake, and why a page reload "fixed" it.

Three layers then turned one slow query into a silent, unbounded hang: the query_events retry loop swallowed up to four 500s back to back, the Prefect client silently retries read timeouts five times (logged at DEBUG), and the e2e haproxy config disables all timeouts.

Non-goals: no change to Prefect-server-wide prepared-statement caching (a global statement_cache_size=0 was considered and rejected as an unbounded performance surface), no changes to Prefect's own query builders.

What changed

• The events/filter endpoint runs SET LOCAL plan_cache_mode = force_custom_plan at the start of its transaction (Postgres only), so its two queries are always planned with the real filter values. Scoped to this transaction — every other Prefect statement keeps prepared-statement plan caching. Cost: ~0.25ms planning per statement.
• The unbounded count(*) — by far the endpoint's most expensive statement — now only runs when the GraphQL query selects count (new include_total input field, default true, so older clients keep the previous behavior). The activity-log UI never selects it, so its page loads skip the aggregate entirely.
• Each query_events retry now logs a warning, so a stalling task manager is visible in the server logs instead of silent.
• dev/knowledge/backend/events.md gains a "Query-path performance constraints" section recording both invariants.

No GraphQL schema changes; the events/filter wire change is backward-compatible in both directions (pydantic defaults/extra-ignore).

How to review

• backend/infrahub/prefect_server/events.py — the SET LOCAL and its scoping rationale.
• backend/infrahub/task_manager/event/query.py — include_total derivation from the GraphQL selection, retry warning.
• backend/infrahub/prefect_server/{models,database}.py — the include_total plumbing.

How to test

Validated on a CI-parity repro stack (testcontainers compose + 5k synthetic events / 40k event_resources) with pg_stat_statements.track_planning=on, 30 sequential requests to /infrahub/events/filter:

	latency per call	plans vs calls
before	~20ms for calls 1–5, then 8.9s from call 6 per pool connection	18 plans / 30 calls (generic-plan reuse)
after	18–20ms flat, all 30 calls	30 plans / 30 calls (custom plan every time)

uv run pytest -c tests/e2e/pytest.ini tests/e2e/activities -m shard_branches_repo

Impact & rollout

• Backward compatibility: include_total defaults to true; old client ↔ new server and new client ↔ old server both keep prior behavior.
• Performance: activity-log page loads drop the unbounded count entirely; the remaining two statements pay ~0.25ms planning each per request; no impact on any other Prefect server query.
• Config/env changes: none.
• Deployment notes: safe to deploy; server and task-manager ship in the same image.

Checklist

• Tests added/updated (validated via live repro stack + pg_stat_statements; no unit harness exists for the Prefect-server extension endpoint)
• Changelog entry added
• Internal .md docs updated (dev/knowledge/backend/events.md)
• I have reviewed AI generated content

🤖 Generated with Claude Code