Bracket IPv6 address literals when rendering URLs and authorities

AD
4mo ago
+216-7

Summary

Working on https://github.com/scrapy/scrapy/pull/7090, it seemed like httpx2 would support IPv6 in general but not through a proxy. Apparently the brackets get lost when proxying, thus the issue.

What Opus wanted me to use a PR description

URL.host and Origin.host hold a host in the form needed to open a connection,
which for IPv6 is the bare address. Four places rendered that bare host into a URL
or an authority without adding the square brackets that an IP-literal requires,
putting malformed values on the wire: ::1:8443 cannot be split back into a host
and a port, and http://::1:8443/ is not a valid URI.

The most visible consequence is that no request to an IPv6 address literal works
through a proxy, in either forwarding or tunnelling mode.

BeforeAfter
CONNECT target (https://[::1]:8443/ via proxy)CONNECT ::1:8443CONNECT [::1]:8443
Request target (http://[::1]:1234/ via proxy)GET http://::1:1234/GET http://[::1]:1234/
Default Host header, no proxyHost: ::1:1234Host: [::1]:1234
bytes(URL("http://[::1]:8080/x"))http://::1:8080/xhttp://[::1]:8080/x
str(Origin(b"http", b"::1", 8080))http://::1:8080http://[::1]:8080

The Host header row applies to httpcore2 used on its own; through httpx2 it is
masked, because httpx2 sets Host itself from the bracketed URL.netloc.

The change

A single helper in _models.py, used at each of those sites:

def format_host(host: bytes) -> bytes:
    return b"[%b]" % host if b":" in host else host

The ":" in host test is the same one httpx2 already uses in
ParseResult.authority and ParseResult.netloc (src/httpx2/httpx2/_urlparse.py),
so both packages now build an authority the same way. A colon cannot occur in a
registered name or in an IPv4 address, so its presence is enough to identify an
IPv6 address. URL.host is bare by construction, since URL.__init__ goes through
urlparse().hostname, which strips brackets.

Origin.host deliberately keeps the bare address: it feeds the socket connect, the
TLS server_hostname, and the SOCKS address, none of which want brackets. Only the
rendering sites changed.

_sync/http_proxy.py is regenerated by scripts/unasync.py, not hand-edited.

Context

This is the bug from encode/httpx#392,
carried over as #145. It was closed in 2019 as not reproducible, correctly at the
time: httpx.URL("https://[::1]:8080").origin.host was then '[::1]', brackets
included. Delegating transport to httpcore, whose Origin.host is bare,
reintroduced it. The same lines are present in encode/httpcore today, and
tests/_async/test_http_proxy.py there has no IPv6 case.

Out of scope, to keep this atomic:

  • #676, the IPv6 zone identifier leaking into the Host header. Different layer
    (httpx2 header rendering rather than httpcore2 authority rendering), and the
    zone still arrives percent-encoded as %25. Both need fixing for link-local
    addresses to work through a proxy.
  • The SOCKS path, where the host is passed to socksio as a protocol address
    field rather than as part of a URL, so the bare form is correct.

Checklist

  • I understand that this PR may be closed in case there was no previous discussion.
    I’ll risk it :slightly_smiling_face:
  • I've added a test for each change that was introduced, and I tried as much as possible to make a single atomic change.
  • I've updated the documentation accordingly.
    No doc changes I assume, this is a bug fix.