Fix HTTP/1.1 response cleanup interrupted by task cancellation

D3
2mo ago
+225-26

Summary

Fixes the HTTP/1.1 response cleanup leak reported in discussion #1200.

If asyncio.Task.cancel() interrupts response close while it waits for the HTTP/1.1 state lock, both stream wrappers have already marked themselves closed. The connection stays ACTIVE and the pool retains its request. Calling aclose() again does nothing, and the next request can raise PoolTimeout with max_connections=1.

As the discussion comments point out, the existing AnyIO shield does not block direct task cancellation. Removing the pool request alone would also leave the connected ACTIVE connection occupying a slot. The fix handles both parts:

  • Close the HTTP/1.1 connection if response cleanup is interrupted, before the pool releases its request.
  • Remove the pool request and reassign waiting requests in finally, even if the inner close raises.
response close interrupted
  -> close HTTP/1.1 connection before releasing its pool request
  -> always remove the pool request and reassign waiting requests
  -> propagate the exception

This happens after connection establishment and response delivery, later in the lifecycle than the case in #982 / PR #983. The synchronous source and tests were regenerated with scripts/unasync.py.

Tests

The regression test cancels close directly with Task.cancel() while the established connection waits for its state lock. It covers unread and fully read response bodies, and checks that cancellation propagates, the old connection is closed and removed, the request list is empty, repeated close is harmless, and a follow-up request succeeds with one connection slot. Separate tests cover trace callbacks raising during HTTP/1.1 and HTTP/2 cleanup in async and sync code. The HTTP/2 test advertises a stream limit of one and verifies that a follow-up request succeeds on a replacement connection.

  • On main 0286b404, .venv/bin/python -m pytest tests/httpcore2/test_cancellations.py -k task_cancellation_during_response_close -q fails both cases because the connection remains ACTIVE. Both pass with this fix.
  • All 18 cancellation tests pass, including the existing AnyIO asyncio/Trio cases.
  • Full suite on Python 3.13.14: 2019 passed, 1 skipped; 100% coverage across 14,557 statements.
  • scripts/check passes: lockfile, formatting, mypy, Ruff, and async/sync generation checks.

Cleanup behavior

Successful close still follows the existing connection reuse path. Cleanup failures now close the affected HTTP/1.1 or HTTP/2 connection and release its pool request. The HTTP/2 guard addresses the review finding that a failed trace callback could otherwise leave a stream permit held on a connection that the pool still considers available. Closing that connection may interrupt other streams sharing it; this applies only when response cleanup raises. Generated sync code follows the same exception handling. There are no public API or data-format changes, and reverting this change requires no migration.

If the fallback transport close also raises, that exception can replace the original cancellation, as in the existing cleanup code. This fix does not guarantee transport closure when the backend itself fails or cleanup receives repeated native task cancellations.

Checklist

  • Previous discussion: #1200.
  • Added regression tests for the cleanup changes.
  • No public API documentation update is needed.