inject observer metrics for rate limiting
Why
The API backpressure layer wired its Prometheus sinks by handing components a mutable observer slot
after construction — PrioritySlotPool.set_observer(), RetryAfterPolicy.set_observer(), and a
ReferenceQueryLoadTracker.set_observer() called from the tracker's own module. That inverted the
dependency direction three ways:
- The primitives owned their sinks.
database/load_signal.pyimporteddatabase/metrics.pyat
module level, so importing the stress-signal primitive the admission gate codes against pulled
Prometheus gauge registration into the import chain. - Consumers mutated injected collaborators.
AdmissionController.__init__reached back into the
pool and the policy it had just been handed to install callbacks defined incontroller.py. The
pool called into its consumer at runtime while the consumer imported it at compile time, and
the wiring was invisible at the application entry point. - One sink per component. A single mutable slot meant two controllers sharing a pool silently
clobbered each other's observer, and no component could have a second sink.
Goal: every component that feeds a metric receives its sinks as a required constructor argument
and pushes values to them, so the concrete sinks are named only where the object graph is wired and
Prometheus stays out of the primitives' import chain.
Non-goals: no change to the shed decision itself — the tiers, thresholds, CoDel behaviour, slot
accounting, Retry-After computation, and every metric name and label are unchanged. This is a
dependency-direction refactor plus configuration hardening, not a behaviour change.
What changed
Behavioral changes
- Non-finite (
inf) values are now rejected for every backpressure float setting. Previously
infpassed thegt=0/ge=1bounds and then crashed startup withOverflowErrorinside
int(base * multiplier), disabled window eviction entirely (horizon = now - inf), or silently
exempted a priority class from shedding (ratio / inf→ tier 0). - The backstop multipliers and shed-stress ratios must now be ordered by priority. A misordered
config used to invert the whole feature — HIGH backstop-rejected while LOW kept queueing, or HIGH
shedding at a lower stress ratio than LOW — and startup accepted it silently. - Worker processes now honour
backpressure_stress_window_seconds. The window was applied by
mutating the tracker inside the admission factory, which only runs on API servers, so any process
without an admission gate kept the hard-coded 20s default and published stress gauges computed
over the wrong window. - Observer failures are contained per observer, on all three components. A raising sink can no
longer corrupt admission state, shed a request that would have been admitted, fail the database
query that fed an observation, or skip the sinks behind it.
Implementation notes
- Four observer protocols, one per component:
AdmissionObserver(four events — offered, admitted,
rejected, sojourn),SlotPoolObserver,RetryPolicyObserver,LoadSignalObserver. Each pushes
values as arguments, so a sink never reads back into the component it observes.AdmissionObserver
is one interface with four events rather than one per metric, because they describe a single
request's passage through one decision. - Concrete sinks live in
api/admission/observers.pyanddatabase/load_signal_metrics.py. Nothing
underapi/admission/outside the entry point imports them:server.pypasses them to
build_admission_controller, which takes them as arguments rather than choosing them. build_admission_controllermoved out ofcontroller.pyinto a newfactory.py. That leaves the
decision logic importing nothing from its own package at runtime.- The process-global tracker moved out of
load_signal.pyintoload_signal_registry.py, which
reads the window from settings at construction.window_secondsis now a read-only property —
resizing a live window would reinterpret the samples already in it. RejectionReasonbecame aStrEnuminconstants.py, so the three reason strings exist in one
place and both the controller and the sink import the type from a leaf module.
What stayed the same
- Metric names, labels, and semantics —
infrahub_admission_*andinfrahub_db_reference_query_*
are byte-for-byte identical, so dashboards and alerts are unaffected. - The
429response envelope,Retry-Aftervalues, excluded paths, and the CORS-preflight bypass. - Default values for every setting. The new constraints only reject configurations that were already
incoherent.
How to review
Suggested order:
slot_pool.py,retry_policy.py,load_signal.py— the primitives. Each gained a protocol and a
requiredobserversargument, and lost itsset_observer. Note the failure guard sits inside
the per-observer loop; moving it outside would silently downgrade isolation from per-observer to
per-event.controller.py— largest diff. Six inline metric calls became four_observe_*methods over the
injected list, each guarding per observer via the_contained_observer_failure()context manager.factory.py,server.py,load_signal_registry.py— the wiring, and the only places that name a
concrete sink.config.py— the new constraints and thevalidate_shed_stress_ratios_ordered_by_priority
validator.- Tests. Recording and failing adapter sinks live in
tests/unit/api/admission/helpers.py; no
mocks.
Worth extra scrutiny:
middleware.py:94still incrementsMISSING_PRIORITY_TOTALdirectly. It is the one metric in
the package written outside a sink — deliberate, since it fires during header parsing before a
controller is involved, but it means the package is not 100% consistent.get_reference_query_load_tracker()now raisesInitializationErrorif settings are unloaded.
Unreachable in practice (the tracker is only reached from the reference query, which requires a
configured database), and chosen over a silent fallback that would diverge from config.infis no longer a way to express "never shed this class." A large finite ratio still is. If
that needs to be first-class, it deserves an explicit representation.
Checklist
- Tests added/updated
- Changelog entry — n/a, API backpressure is unreleased and this refactors it before ship
- External docs updated — n/a, the generated config reference is unchanged (constraints do not
appear in it) - Internal .md docs updated (
dev/knowledge/backend/api-backpressure.md) - I have reviewed AI generated content
Summary by cubic
Inject metrics via observers across the rate limiting path to decouple Prometheus sinks from admission logic and keep failures contained. Moves controller wiring to a factory, shares the DB load-signal via a registry, and tightens backpressure config validation.
- Refactors
- Introduced observer interfaces:
AdmissionObserver,RetryPolicyObserver,SlotPoolObserver,LoadSignalObserver. - Added metric sinks:
AdmissionMetricsObserver,SlotPoolMetricsObserver,SustainedLoadMetricsObserver,LoadSignalMetricsObserver. AdmissionController,PrioritySlotPool,RetryAfterPolicy, andReferenceQueryLoadTrackernow takeobserversand isolate sink failures.- Moved
build_admission_controllertofactory.py;server.pywires observers at startup. - New
load_signal_registry.pyprovides the shared tracker;load_signal_metrics.pypublishes its gauges; window length comes from settings. - Replaced string reasons with
RejectionReasonenum (used as thereasonmetric label). stress_shed_fraction()now takes atier; tests updated accordingly.- Backpressure settings guarded: shed stress ratios must be ordered by priority, non‑finite floats are rejected, and backstop multipliers are bounded (
low≤ 1 ≤high).
- Introduced observer interfaces: