PR #29 - ArchiveBox/abx-plugins

Status: draft, needs more testing before merge

I'm opening this as a draft because the plugin has not been exercised
inside a live ArchiveBox install yet — only against the standalone
hook contract (the abx-plugins repo's test harness + manual runs
against a handful of real sitemaps). Before this is ready to land I'd
like to:

run it end-to-end inside an actual archivebox add flow and confirm
the emitted Snapshot records feed the host's crawl frontier as
expected
test the depth/limit interaction with the host's own max_depth /
max_urls settings on a real install
try it on more varied sites, especially SPAs that only publish a
partial sitemap (pairs with parse_dom_outlinks)
get feedback on whether the new MAX_SITEMAPS and
ALLOW_PRIVATE_HOSTS knobs hit the right defaults for the
ecosystem

If anyone on the abx-plugins side can spot something that won't survive
the host integration, I'd rather hear it now than after I've put more
hours into polish. Happy to keep iterating.

What this does

Adds a sitemap-driven URL discovery hook so a single seed URL can
expand into a full-site crawl without an external crawler in the
loop. Closes the long-standing
ArchiveBox/ArchiveBox#191.

The host (ArchiveBox or abx-dl) still owns crawl ordering, dedup,
and the global max_depth / max_urls caps — this plugin only emits
Snapshot JSONL records, so existing crawl semantics are preserved.

Features

Capability	Notes
Standard `urlset` parsing	Preserves `<lastmod>` as `bookmarked_at`, also reads `<priority>` and `<changefreq>`.
`sitemapindex` recursion	Configurable depth + total-fetch cap, cycle-safe via visited-set.
Gzip handling	Magic-byte sniff, `Content-Encoding: gzip` header, `.gz` URL suffix.
BOM-tolerant XML	UTF-8, UTF-16 LE, UTF-16 BE.
`robots.txt` `Sitemap:` discovery	Default on; handles multiple directives per file.
Fallback path probing	`/sitemap.xml`, `/sitemap_index.xml`, `/sitemap-index.xml`, `/wp-sitemap.xml`, `/sitemap.xml.gz` (configurable).
Include / exclude regex filters	With a scan-length cap to blunt catastrophic-backtracking risk.
`same_host_only` mode	Also applied to child sitemap fetches (per sitemaps.org §2.2).
Priority + changefreq filters	`PRIORITY_MIN` threshold with optional `REQUIRE_PRIORITY` strictness; `CHANGEFREQ_ALLOWED` allowlist.
Sort modes	`url` / `lastmod` / `priority` / `order` (sitemap order).
Image / video / news extensions	Optional. Media extras carry `tags=sitemap-media` and pass through the same URL policy as page URLs; news emits `Tag` records for publication names.
HTTP retry + backoff	408, 429, 5xx, network errors; configurable count and base delay.
Bounded redirects	Hard cap on redirect count, rejects non-HTTP schemes and private hosts unless `ALLOW_PRIVATE_HOSTS=true`.
TLS verification toggle	Falls back to `CHECK_SSL_VALIDITY`.
Custom User-Agent + Accept-Language	Per-plugin override with global fallback.
Schemeless URL handling	`//host/path` resolved against the sitemap's own scheme.
Fragment stripping	`#anchor` variants don't produce duplicate snapshots.
`MAX_URLS` cap	Default 5000; applies before host's frontier caps.
`MAX_SITEMAPS` cap	Default 100; counts attempts, not successes, so an adversarial index can't pivot into thousands of fetches.
Verbose mode	One `fetching sitemap …` line per fetch on stderr.

No new binary dependencies. Stdlib (xml.etree, gzip, ssl,
urllib) plus the existing rich_click, defusedxml, and
abx_plugins.plugins.base.utils.

Security posture

Untrusted sitemap input is the default threat model. Defenses:

XML hardening via defusedxml.ElementTree.iterparse. DTDs and
entities are refused; billion-laughs and XXE fail-closed.
Response size cap before parsing (default 50 MiB).
Decompression cap + ratio cap (default 200 MiB and 100×) so
gzip bombs fail with status=failed rather than OOM.
Scheme allowlist for emitted URLs: only http and https;
javascript:, data:, ftp:, etc. are dropped.
file:// containment: only allowed when the seed itself is
file:// or the operator opts in.
Bounded, validated redirects via a custom
HTTPRedirectHandler that caps the chain and refuses non-HTTP
schemes and private (loopback / RFC1918 / link-local / multicast)
targets unless explicitly allowed.
Sitemap attempt cap (MAX_SITEMAPS) defeats adversarial
sitemap-indexes pointing at thousands of broken children.
Streaming parse via iterparse with elem.clear() plus
root.remove(elem) keeps resident XML bounded to a single
<url> subtree even on 500k-URL documents.
DNS-rebinding caveat documented in the README — the
private-host check re-resolves on each policy call but cannot pin
the IP through to urllib's connect-time lookup. If your threat
model includes rebinding, run behind an outbound firewall.

Hook contract

Standard on_Snapshot__* per abx-plugins/README.md:

stdout: 0+ Tag records, 0+ Snapshot records, terminal
ArchiveResult. Every non-empty stdout line is a JSON record (the
test harness asserts this).
stderr: discovery / fetch error lines and the human summary.
succeeded if any URL emitted, noresults when filters or empty
sitemaps prune everything, skipped when the plugin is disabled,
failed only when every candidate sitemap is unfetchable /
unparseable or a security guard tripped.
SNAP_DIR/parse_sitemap_urls/urls.jsonl written atomically and
removed on noresults / failed.

Summary string carries counters so logs make it obvious why nothing
emitted:

0 URLs parsed (visited 1 sitemap(s); skipped_filter=3 skipped_host=0
 skipped_priority=2 skipped_changefreq=0 skipped_scheme=1
 skipped_extras=0)

Filename

Numbered 76 — sits after parse_dom_outlinks (75) and before any
later snapshot work. The 73 slot is already taken by
parse_netscape_urls.

What I tested

ruff check        → All checks passed
ruff format       → 5 files already formatted
pyright           → 0 errors, 0 warnings
pytest            → 83 passed in ~80s

83 tests across two files:

tests/test_parse_sitemap_urls.py (34) — basic urlset and
sitemapindex parsing, gzip, filtering, malformed input, robots.txt
discovery, HTTP fetch via pytest-httpserver, ordering and dedup.
tests/test_parse_sitemap_urls_advanced.py (49) — BOM stripping,
unicode, schemeless URLs, priority and changefreq filtering, all
four sort modes, image / video / news extensions, HTTP retry on
5xx, retry exhaustion, redirect following, redirect count cap,
scheme allowlist, remote → file:// chain refused, billion-laughs
blocked, external entity blocked, gzip bomb capped, response body
capped, truncated gzip handled cleanly, fragment stripping, media
extras subject to full URL policy, MAX_SITEMAP_DEPTH
semantics (0 walks only seed, 1 walks one child level),
MAX_SITEMAPS enforced on attempts (not just successes),
cross-host child sitemap refused under SAME_HOST_ONLY, JSONL
contract enforced, 500k-URL streaming bounded by wall time.

Manual smoke runs against real sites (all clean):

Site	Setup	Result
docs.python.org	root URL discovery via robots.txt	8 URLs
github.blog	WP sitemap-index	200-URL cap respected, 2 sitemaps walked, `bookmarked_at` populated
www.sitemaps.org	multi-locale gzipped sitemap	20 URLs across `/en/`, `/es/`
bbc.com news archive	`SORT_BY=lastmod`	15 newest-first articles
github.blog	`SAME_HOST_ONLY=true`	10 URLs, all `github.blog`
docs.python.org	`EXCLUDE_REGEX=/3\.1[0-2]/`	5 / 8 URLs (3 filtered)
sitemaps.org	`PRIORITY_MIN=0.8`	0 / 84 (no priority hints declared)

Compatibility

New plugin, no existing behavior changes.
No required_plugins, no required_binaries, no preflight impact.
Pure stdlib + existing dev deps; the only new runtime dependency is
defusedxml>=0.7.1 for XML hardening.

Open questions for reviewers

Default for MAX_SITEMAPS is 100 — reasonable, or should it be
higher / lower? Tied to typical real-world sitemap-index fan-out.
ALLOW_PRIVATE_HOSTS defaults to false, which surfaces a
private_host error when someone tries to archive their own
intranet without opting in. Discoverable enough, or should the
default flip when the seed itself is private?
The plugin pairs naturally with parse_dom_outlinks for SPAs.
Worth dropping a one-liner in parse_dom_outlinks/README.md
recommending sitemap discovery as the "first pass" of a full-site
crawl?
Anything in the hook contract I'm reading wrong from
abx-plugins/README.md?

Marking ready-for-review only after the e2e run inside ArchiveBox is
clean.

Summary by cubic

Add parse_sitemap_urls plugin to discover URLs from sitemap.xml and emit Snapshot records, enabling full‑site crawls from a single seed without an external crawler. Preserves host crawl ordering, dedup, and depth/URL caps.

New Features
- Parse urlset and sitemapindex (recursive, cycle-safe); handles gzip and BOM.
- Discover via robots.txt and fallback probing; resolves schemeless URLs; strips fragments.
- Filters: same-host-only, include/exclude regex, priority/changefreq; sort by url/lastmod/priority/order.
- Safety: hardened XML via defusedxml, response/decompression caps, HTTP‑only allowlist, bounded redirects, private-host gate.
- Controls: hard caps on URLs and sitemap fetches, retry/backoff, TLS toggle, custom UA/Accept‑Language.
- Outputs JSONL Snapshot (and Tag for news); preserves <lastmod> as bookmarked_at.
Dependencies
- Add defusedxml>=0.7.1. No new binaries.