Bracket IPv6 address literals when rendering URLs and authorities
Summary
Working on https://github.com/scrapy/scrapy/pull/7090, it seemed like httpx2 would support IPv6 in general but not through a proxy. Apparently the brackets get lost when proxying, thus the issue.
What Opus wanted me to use a PR description
URL.host and Origin.host hold a host in the form needed to open a connection,
which for IPv6 is the bare address. Four places rendered that bare host into a URL
or an authority without adding the square brackets that an IP-literal requires,
putting malformed values on the wire: ::1:8443 cannot be split back into a host
and a port, and http://::1:8443/ is not a valid URI.
The most visible consequence is that no request to an IPv6 address literal works
through a proxy, in either forwarding or tunnelling mode.
| Before | After | |
|---|---|---|
CONNECT target (https://[::1]:8443/ via proxy) | CONNECT ::1:8443 | CONNECT [::1]:8443 |
Request target (http://[::1]:1234/ via proxy) | GET http://::1:1234/ | GET http://[::1]:1234/ |
Default Host header, no proxy | Host: ::1:1234 | Host: [::1]:1234 |
bytes(URL("http://[::1]:8080/x")) | http://::1:8080/x | http://[::1]:8080/x |
str(Origin(b"http", b"::1", 8080)) | http://::1:8080 | http://[::1]:8080 |
The Host header row applies to httpcore2 used on its own; through httpx2 it is
masked, because httpx2 sets Host itself from the bracketed URL.netloc.
The change
A single helper in _models.py, used at each of those sites:
def format_host(host: bytes) -> bytes:
return b"[%b]" % host if b":" in host else host
The ":" in host test is the same one httpx2 already uses in
ParseResult.authority and ParseResult.netloc (src/httpx2/httpx2/_urlparse.py),
so both packages now build an authority the same way. A colon cannot occur in a
registered name or in an IPv4 address, so its presence is enough to identify an
IPv6 address. URL.host is bare by construction, since URL.__init__ goes through
urlparse().hostname, which strips brackets.
Origin.host deliberately keeps the bare address: it feeds the socket connect, the
TLS server_hostname, and the SOCKS address, none of which want brackets. Only the
rendering sites changed.
_sync/http_proxy.py is regenerated by scripts/unasync.py, not hand-edited.
Context
This is the bug from encode/httpx#392,
carried over as #145. It was closed in 2019 as not reproducible, correctly at the
time: httpx.URL("https://[::1]:8080").origin.host was then '[::1]', brackets
included. Delegating transport to httpcore, whose Origin.host is bare,
reintroduced it. The same lines are present in encode/httpcore today, and
tests/_async/test_http_proxy.py there has no IPv6 case.
Out of scope, to keep this atomic:
- #676, the IPv6 zone identifier leaking into the
Hostheader. Different layer
(httpx2header rendering rather thanhttpcore2authority rendering), and the
zone still arrives percent-encoded as%25. Both need fixing for link-local
addresses to work through a proxy. - The SOCKS path, where the host is passed to
socksioas a protocol address
field rather than as part of a URL, so the bare form is correct.
Checklist
- I understand that this PR may be closed in case there was no previous discussion.
I’ll risk it :slightly_smiling_face: - I've added a test for each change that was introduced, and I tried as much as possible to make a single atomic change.
- I've updated the documentation accordingly.
No doc changes I assume, this is a bug fix.