Track connection pool state incrementally instead of scanning on every request

KL
2mo ago
+1264-244

Summary

AsyncConnectionPool._assign_requests_to_connections ran on every enqueue and every release and walked every pooled connection each time (several delegated state queries per connection, plus a poll() on every idle socket via has_expired()), so per-request cost grew with the number of connections. With an unbounded pool at 512 concurrent requests, ~85% of client CPU was pool bookkeeping and throughput fell to a third of the c16 figure.

The pool now keeps its state incrementally and a pass is O(1) in the common case:

  • Connections and in-flight requests live in insertion-ordered dicts; requests waiting for a connection are a FIFO deque.
  • Idle HTTP/1.1 connections are indexed by origin, oldest first, which serves reuse, max_keepalive_connections eviction (longest idle first, as before) and the "close an idle connection to another origin when the pool is full" case.
  • Sharable connections (HTTP/2, or not yet negotiated with http2=True) are kept per origin and checked with is_available() on lookup, so a burst against a warm HTTP/2 connection is still multiplexed within one pass.
  • Reservations are refcounted per connection (the idea from #1076), and a pass re-examines only the connections released since the previous pass.
  • A server-closed idle socket is detected when the connection is about to be reused (one probe per reuse) instead of probing every idle socket on every pass; keepalive expiry is swept only once the oldest idle connection may have expired.
  • Origin gets a __hash__ consistent with its __eq__, and AsyncPoolRequest creates its wakeup event only when it actually has to wait (the common path assigns before waiting, so no anyio.Event and no sniffio call per request).

_assign_requests_to_connections keeps its name and cadence (once per enqueue, once per release), so the existing pass-counting tests hold. #1076 becomes redundant with this change.

Benchmarks

scripts/benchmark --lib httpx2 --python main=... --python pool=..., CPython 3.14 + zuvloop, 2 vCPU, interleaved rounds (medians):

scenariomainthis PR
c1/1k798 rps · 391 us/req817 rps · 371 us/req
c16/1k2,619 rps · 299 us/req3,104 rps · 251 us/req
c128/1k2,037 rps · p99 90 ms · 426 us/req3,494 rps · p99 59 ms · 248 us/req
c512/1k787 rps · p99 1152 ms · 1058 us/req3,267 rps · p99 218 ms · 263 us/req
c64/100k2,023 rps · 412 us/req2,695 rps · 317 us/req
c64/1k POST2,266 rps · 379 us/req2,974 rps · 287 us/req
c8/5m, c8/1m POSTunchanged (body-bound)unchanged

Retention (rps at c512 over rps at c16) goes from 30% to 105%; per-request CPU is now flat across concurrency. Measured on the final state of the branch, after the review changes.

Tests

New tests cover: an idle connection closed by the server is discarded on reuse; staggered keepalive expiry keeps younger idle connections (HTTP/1.1 and HTTP/2); an externally closed HTTP/2 connection is discarded; and the AsyncEvent/Event primitives directly, since the pool no longer exercises every branch on each request. The async tests use anyio.sleep, mapped to a threaded concurrency.sleep by unasync.

Checklist

  • I understand that this PR may be closed in case there was no previous discussion. (This doesn't apply to typos!)
  • I've added a test for each change that was introduced, and I tried as much as possible to make a single atomic change.
  • I've updated the documentation accordingly.

🤖 Generated with Claude Code