fix(limits): isolate in-flight Redis I/O on a dedicated thread and partition fallback

Merged
RU
7h ago
+781-216

Problem

  1. Main Tokio runtime starvation during request bursts: ConnectionManager's socket pipeline task, claim EVAL deadlines, per-slot renewal tasks, release retries, and reconnect handshakes all ran on the serving Tokio runtime. On a CPU-capped replica (e.g. a 2-worker runtime), bursts of large JSON request bodies occupy the worker threads with synchronous parsing long enough that Redis replies sit unread past the 250 ms claim deadline — even when Redis itself answers in tens of microseconds. Three such timeouts drop the healthy connection and trigger a TLS reconnect on the same congested runtime, causing multi-second fallback windows.
  2. Unpartitioned fallback and split counters: When falling back locally, each replica admitted up to the full global limit rather than its share across active replicas, did not count slots already admitted via Redis on that replica, and did not register fallback-admitted slots back into Redis once the connection recovered.

Change

  • Dedicated "inflight-redis" OS thread: SharedRedis::connect spawns a dedicated OS thread with its own single-threaded Tokio runtime that owns ConnectionManager and executes all claims, 250 ms deadlines, withdrawals, release retries, reconnect loops, batched renewals, and pod heartbeats off the HTTP worker pool. Straggler timeouts/errors on an already-replaced connection no longer tear down the replacement connection.
  • Unified per-replica counter and scoped partitioned fallback: Every admitted slot on a replica (both Redis-admitted and fallback-admitted) is tracked in a single PodInflightCounter. Each replica lazily heartbeats only the limiter scopes it actually serves ("realtime" vs "batch", 5 s interval / 15 s lease) so API replicas and batch daemon replicas do not inflate each other's replica counts, and local fallback caps concurrency at limit.div_ceil(live_pods).
  • Batched renewals and reconnect re-registration: Replaces per-slot renewal tasks with a single 30 s batched SYNC_KEY_SCRIPT pass (chunked at 512 members per key). Active fallback slots are immediately synced to Redis upon reconnect so global counts stay accurate across failovers and recoveries; fallback slots that finish before ever reaching Redis drop with zero Redis traffic.

Tests

  • claims_complete_when_caller_runtime_workers_block: freezes all threads of a 2-worker caller Tokio runtime for 2 × REDIS_TIMEOUT while a claim is in flight and verifies the dedicated thread completes the claim with 0 timeouts and 0 withdrawals.
  • fallback_splits_the_limit_across_live_pods_per_scope: verifies "realtime" and "batch" heartbeats track live replicas independently and fallback divides limit across live replicas in the same scope.
  • fallback_counts_slots_already_admitted_through_redis: verifies slots admitted via Redis count toward the replica's partitioned fallback cap when Redis disconnects.
  • fallback_slots_sync_to_redis_on_reconnect_and_release_on_drop: verifies slots admitted locally during an outage sync into Redis upon reconnect and send ZREM when dropped.
  • All 13 unit tests in realtime_inflight::tests and cargo clippy pass with zero warnings.