fix(webfetch): block SSRF to metadata, loopback and private ranges
Merged
TE
2d ago+2479+2479+0123456789012345678901234567890123456789-85-85-01234567890123456789
Purpose
web_fetch only checked the URL scheme, so it could reach cloud metadata, loopback (including Denkeeper's own API) and private ranges, directly or through a redirect. Found in the OWASP Agentic Top 10 review (ASI02).
Closes #589
What a reviewer needs to know
- Checked at connect time. A new
internal/netguardpackage refuses blocked addresses in the dialer, so redirects, robots.txt lookups and DNS rebinding are covered. A URL-string check runs first only for a clearer error. - Closed by default, two opt-ins.
[web.fetch] allow_loopbackandallow_privateopen their own ranges. Metadata (incl. Azure168.63.129.16), link-local and the proxy's own address never open. The docs spell out how broadallow_privateis. - Blocked URLs never reach Jina. The fallback used to fire on any error. A robots.txt or agents.txt disallow still falls back to Jina, as before.
- Local-use NAT64
64:ff9b:1::/48counts as private. A host whose DNS64 uses it needsallow_privateto fetch IPv4-only sites. - Proxy kept. With
HTTP(S)_PROXYset, the proxy must enforce egress. Documented once under "Network policy". allowed_domains/blocked_domainsare name-only. Bad entries (.evil.com, IP literals, unmappable hosts) fail at load.- Remote MCP is untouched. Its guard allows RFC1918 on purpose for LAN servers.
- Browser allowlist now uses
netguard, but has no production caller. The container's own check is Temikus/denkeeper-browser#89.
Test plan
-
just hook - Tests use live
httptestservers and the real dialer. The redirect, metadata, IDNA, proxy and default-config tests fail onmain
š¤ Generated with Claude Code
https://claude.ai/code/session_01YGMCH6b8gjLdh3ZaC2QwRh
Summary by cubic
Blocks web_fetch, the browser allowlist, and the Jina fallback from reaching cloud metadata, loopback, and private ranges, directly or via redirects. The check runs on the IP each connection reaches at dial time, so redirects, robots/agents.txt lookups, and DNS rebinding are covered. Closes #589.
The guard
- New
internal/netguardpackage classifies addresses in the dialer; loopback and private ranges open only via[web.fetch] allow_loopbackandallow_private. - Metadata (incl. Azure
168.63.129.16, AWSfd00:ec2::254, Alibaba100.100.100.200), link-local, unspecified, multicast, and broadcast never open, even inside private ranges. - IPv6 forms carrying an IPv4 address (mapped, NAT64, 6to4) are judged by that IPv4; local-use NAT64 layouts are checked at every RFC 6052 position.
Behavior changes
- A blocked URL never reaches Jina, and Jina's client is guarded too, checking every redirect hop by URL; refusals name the setting that would open the class.
allowed_domainsandblocked_domainsmatch the IDNA-mapped host; IP literals, numeric last labels, and malformed entries fail at config load. They cover the URL sent to Jina, not Jina's own redirect targets, and robots/agents.txt disallows still fall back to Jina.- The proxy dial is exempt, so the proxy must enforce egress; a direct request to the proxy's own address is refused, and
NO_PROXYhosts go through the address check. - Empty-label and non-canonical numeric hosts are refused so a pure-Go resolver failure cannot forward them to Jina.
- The browser allowlist uses the same classifier, IDNA-maps hosts and patterns, and refuses unmappable patterns; the remote-MCP guard is unchanged and still allows RFC1918.
- Enabling an opt-in logs a single startup warning, and a reload that changes an opt-in logs a warning that running agents keep the old policy until restart.