feat(api): priority-aware server-side API backpressure

Merged
DG
4mo ago
+6983-57

Why

When an Infrahub instance runs heavy background work (generators, artifacts, diffs, repo syncs, computed attributes) while a human uses the frontend, both compete for the same finite uvicorn worker pool and Neo4j connection pool. Today the API is origin-blind — frontend and background requests are indistinguishable — so under heavy background load the API can no longer serve the frontend and the app appears to hang.

Goal: keep the frontend responsive under background overload by shedding load by priority, with no per-customer tuning.

Non-goals (out of scope, tracked separately): the client/SDK backoff that consumes Retry-After, the frontend setting X-Priority: high, and enforcement that a high claim is legitimate. This PR is server-side only.

Implements Jira IFC-2886 (which implements JPD idea INFP-636).

What changed

Behavioral changes (what operators/callers observe):

  • Requests may declare priority via a new X-Priority header (high / normal / low); missing or invalid → normal, so the layer is safe to deploy before any caller is updated.
  • Under overload the server sheds low-priority requests first and protects high-priority (interactive) requests, returning a fast 429 Too Many Requests + Retry-After with no handler work performed. Shedding is adaptive (CoDel on measured slot-wait / sojourn time), not a fixed threshold, and self-terminates within a bounded window after overload ends.
  • Eight new per-priority metric families on the existing /metrics endpoint (infrahub_admission_*): offered, admitted, rejected-by-reason (codel/backstop), in-flight, waiters, sojourn histogram, derived max_concurrency, and a no/invalid-priority counter for adoption tracking.
  • New config knobs (INFRAHUB_API_BACKPRESSURE_*) including a kill-switch (backpressure_enabled, default on) and a new INFRAHUB_DB_MAX_CONNECTION_POOL_SIZE (default 100, now passed explicitly to the Neo4j driver).

Implementation notes:

  • New package backend/infrahub/api/admission/: a pure-ASGI AdmissionMiddleware (registered outermost), a cancellation-safe PrioritySlotPool (per-class FIFO, priority hand-off, modelled on asyncio.Semaphore), a pure CoDelController with an injected clock, an AdmissionController composing them, capacity derivation, and the metrics module.
  • Per-worker and coordination-free by design (no shared/global limiter, no Redis). max_concurrency is derived from the process's own Neo4j pool size — no hard-coded constant.

What stayed the same:

  • No database schema or migration change; no GraphQL change; no persistence.
  • No new hard dependency (prometheus-client was already present transitively and imported in-tree; it is now declared explicitly).
  • Default behaviour is unchanged under normal load: with no caller sending X-Priority, everything is normal and nothing is shed until load exceeds the derived cap.

Suggested review order

  1. contracts/x-priority-header.md and contracts/metrics.md in the spec dir — the external contract.
  2. slot_pool.py and codel.py — the concurrency core (cancellation safety, CoDel state machine).
  3. controller.py → middleware.py → server.py — how they wire together.
  4. Tests under backend/tests/unit/api/admission/ and backend/tests/component/api/test_admission_middleware.py.

Related Context

Full spec, plan, research, data model, contracts, critique, and implementation report live in dev/specs/ifc-2886-priority-api-backpressure/. Key success criteria: high shed rate ≈ 0% under sustained overload while low sheds first (SC-002); sub-interval bursts shed zero (SC-003); every shed is a fast 429 + Retry-After with no handler work (SC-004); shedding self-terminates after overload (SC-005).

The headline latency bound (SC-001) is intentionally discovery-measured against a live overload scenario (see Test Plan) rather than pre-committed.

Documentation Updates

  • Added changelog fragment changelog/+ifc-2886.added.md.
  • Regenerated docs/docs/reference/configuration.mdx for the new settings.

How to review / test

# Deterministic unit tests (fake-clock CoDel, slot-pool cancellation, capacity, parser)
uv run pytest backend/tests/unit/api/admission/ -v

# Component tests (middleware end-to-end: classification, 429+Retry-After, gradient, metrics, kill-switch, excluded paths)
uv run pytest backend/tests/component/api/test_admission_middleware.py -v

Both suites pass locally with no external infrastructure (42 tests). CI to watch: backend unit/component, ruff/mypy lint, and validate-generated-documentation (the regenerated config reference is committed).

Live overload / SC-001 discovery (manual, needs a running server):

export INFRAHUB_DB_MAX_CONNECTION_POOL_SIZE=4
export INFRAHUB_API_BACKPRESSURE_CODEL_TARGET_SECONDS=0.005
export INFRAHUB_API_BACKPRESSURE_CODEL_INTERVAL_SECONDS=0.1
# drive a saturating `X-Priority: low` stream + a steady `X-Priority: high` stream, then:
curl -s localhost:8000/metrics | grep infrahub_admission_
# expect rejected_total{priority="low"} climbing while {priority="high"} stays ≈ 0

Impact & rollout

  • Backward compatibility: fully compatible. No X-Priority header → normal, no behaviour change under normal load. No request/response body change for non-shed requests.
  • Performance: protects contended API/Neo4j capacity under load; hot-path cost is a header read + slot attempt + CoDel eval (O(1), per-worker).
  • Config/env changes: new INFRAHUB_API_BACKPRESSURE_* settings and INFRAHUB_DB_MAX_CONNECTION_POOL_SIZE (default 100 preserves current driver behaviour).
  • Deployment notes: safe to deploy. Ships inert under normal load; kill-switch INFRAHUB_API_BACKPRESSURE_ENABLED=false is the instant rollback. Interactive protection is fully realized once the frontend sends X-Priority: high (separate ticket); until then interactive traffic is normal.
  • Governance heads-up: new X-Priority request header + 429 behaviour (public API surface); the middleware sits in the admission path and reads a client-controlled header (borderline auth, no auth logic changed); X-Priority is untrusted and cooperative in v1 (enforcement deferred).

Checklist

  • Tests added/updated (42 unit + component tests)
  • Changelog entry added (changelog/+ifc-2886.added.md)
  • External docs updated (regenerated configuration.mdx)
  • Internal .md docs updated (spec/plan/contracts/report under dev/specs/ifc-2886-priority-api-backpressure/)
  • I have reviewed AI generated content

🤖 Generated with Claude Code


Summary by cubic

Adds priority-aware, server-side API backpressure with CoDel- and database‑stress–driven shedding and adaptive Retry-After to keep interactive requests responsive under load. The per-worker cap derives from the DB pool with a default concurrency factor of 0.5 (pool × 0.5), enabled by default behind INFRAHUB_API_BACKPRESSURE_ENABLED.

  • New Features

    • Priority admission (high/medium/low), exact excluded-path matching, CORS preflight bypass, and CORS allow‑list for x-priority; per‑priority infrahub_admission_* metrics; prometheus-client declared.
    • Database stress-aware shedding from a rolling median of the reference permission query with tiered, graduated fractions (reason="stress"), alongside CoDel; new INFRAHUB_API_BACKPRESSURE_* knobs; DB pool default 100.
    • Adaptive Retry-After scaled by shed intensity and sustained-load duration (new sustained-load metric), capped under the SDK per‑retry ceiling.
    • Client integration: the frontend sends X-Priority (high by default; low opt‑in; preserved on retries/uploads; only for the Infrahub API origin). Background flows set SDK RequestContext.priority; artifact generation no longer forces HIGH. The python_sdk tracks infrahub-develop with a 429 retry budget of 10.
  • Bug Fixes & Refactors

    • Slot‑pool observer failures can’t leak slots or strand waiters; non‑finite concurrency factor is rejected at config load.
    • Admission controller is built at app startup and read from app.state; excluded paths and CORS preflights bypass admission before reading startup state; exclusions match exact path or descendant and include /api/config; middleware decision branching is exhaustive.
    • CI: disable backpressure only for the legacy Playwright E2E job via INFRAHUB_API_BACKPRESSURE_ENABLED=false; default remains enabled elsewhere.
    • Internal cleanup: RetryAfterPolicy now pushes sustained‑load duration to a registered observer (metrics‑free, sink pattern), wired in the controller; renamed codel map to codel_priority_map for clarity.
    • Docs: extracted IFC‑2886/IFC‑2890 into ADRs and backend/frontend knowledge; archived specs as “extracted” and fixed index links and an archived‑spec knowledge link depth; no behavior change.

Update since original description

  • Tier rename normal → medium. Header values are now high / medium / low; a missing or invalid value resolves to medium. Read every normal above as medium.
  • Database-stress signal added. The admission decision now also consumes a per-worker signal that times the global permission (reference) query: past per-class stress-ratio triggers it sheds a graduated fraction of a class, reported as reason="stress" alongside codel/backstop. This augments the CoDel sojourn signal described above.
  • Config defaults retuned. DB connection pool 100 → 50, admission concurrency factor 1.0 → 0.5, plus new INFRAHUB_API_BACKPRESSURE_* stress knobs.
  • New backend knowledge doc dev/knowledge/backend/api-backpressure.md documents the admission layer and the stress signal (the IFC-2886 spec predates the stress signal and the rename).