PR #201 - firecrawl/anydoc

CHM manuals currently fail format detection, so they cannot use anydoc's file/bytes APIs or shared document model. This draft adds a native CHM frontend across Rust, Node.js, Python, WebAssembly, and the CLI.

Refs #200.

Implementation

Detect ITSF and .chm, then route Format::Chm through the existing dispatcher, shared Document model, and Markdown serializer.
Read ITSF/ITSP/PMGL structures in memory. Support 4/8-byte ResetTable offsets and sequential SpanInfo streams when no ResetTable is available. Decode the solid section once and share it through range views; stored entries borrow the input. Preserve entry-count, entry-size, total-byte, and asset budgets without charging shared section bytes again for each entry.
The private LZX decoder is adapted from lzxd 0.2.7, with its MIT license and source notice under src/formats/chm/lzx/. The published dependency does not expose consumed input bytes; the local adaptation returns this for each frame so SpanInfo decoding does not guess boundaries or replay the stream. It preserves unfinished blocks and dictionary/tree state, handles padding and E8 positions across resets, and rejects invalid match offsets.
Decode HTML using BOM/charset declarations, then the book's Windows language encoding, including GBK/Big5 and Windows-1250 through Windows-1258. ASCII pages and HHC files retain the book's encoding for legacy percent-escaped URLs. CSS honors BOM/@charset and the referring page's encoding.
Adapt recovered HTML with bounded html5ever parsing into the existing Element tree and shared converter. Weak sibling links provide constant-time insertion/removal during HTML recovery; a flat arena owns nodes and avoids recursive sibling-chain destruction. Resolve HTML base href for links/images/stylesheets while retaining actual topic paths for anchor definitions. Preserve the topic title when the body contains only subheadings.
Preserve HHC hierarchy as nested lists, convert topics once in TOC order, and append unlisted topics in stable path order. Retain referenced images through AssetSink. Shared HTML anchors and tight-list spacing use the existing serializer.

Discussion points / current boundaries

The container reader and LZX adaptation are private to this frontend. html5ever is the only new registry dependency; encoding_rs was already present.
Decoding the entire solid section trades resident memory for bounded decompression work. The local validation file expands to about 111 MB. Topic views share that allocation, and the fixed byte budget counts the solid section once plus independently accessed stored entries.
Binary TOCs are outside this first pass. Missing, unreadable, or binary-only TOCs use deterministic path order.
Embedded image bytes are available in Document.assets; Markdown follows the existing alt-text behavior.
A CHM fuzz target is included, but no sustained fuzz campaign has been run.

Validation

Regression tests were exercised failing before the corresponding fixes. Nine small, original CHM fixtures are reproducible with python3 tests/fixture-src/chm.py. They cover legacy URLs/CSS/LCIDs, HTML bases, topic titles, and a real Huffman-compressed four-frame SpanInfo stream with dictionary dependencies and resets. The external manual and full output are excluded from Git.

cargo test --locked: 397 passed, two external-corpus tests ignored by default. Existing snapshots are unchanged; new fixtures have new snapshots.
Node: release build and 22 tests passed; generated bindings are unchanged.
Python: abi3 release wheel and 13 tests passed on Python 3.12.
WebAssembly: release web build, wasm-opt, and 10 tests passed. All bindings exercise the multi-frame Huffman/SpanInfo fixture.
Formatting, workspace Clippy (--all-targets --all-features -- -D warnings), and wasm-target Clippy passed. Cargo packaging includes the private decoder sources and license.
The CHM benchmark runner has five passing accounting/routing tests; bench/convert.py --formats chm accepts all nine original fixtures. bench/chm.py measures per-process peak RSS and conversion/wall time on macOS/Linux, with external input paths and no converted contents saved.
Performance regressions were tested failing first: stored/solid entry copies and duplicate budget accounting, plus repeated wide-sibling insert/remove operations. Additional HTML tests cover moves, text merging, reparenting, templates, comments, and wide-tree destruction.
Local manual: 17,556,247-byte input, all 15,527 HTML topics retained, 4 referenced assets, 30,929,290-byte Markdown without Unicode replacement characters. Rerun with ANYDOC_CHM_SAMPLE=/path/to/manual.chm cargo test --test chm local_chm_manual -- --ignored --nocapture.

Performance validation

Release builds on the same 10-core, 24 GiB Mac, with serial fresh-process comparisons against the previous implementation (three runs, medians; OS caches warm after the first conversions):

Input / metric	Before	After
Local manual, CHM to Markdown API time	2.551 s	2.481 s
Local manual, process peak RSS	535.8 MiB	450.0 MiB
Synthetic CHM with 128,000 foster-parented divs	2,018 ms	76 ms

Large-manual conversion time is effectively unchanged within observed variation; peak RSS fell by about 86 MiB. Synthetic sibling-insertion recovery scales approximately linearly after the change. The manual's 30,929,290-byte Markdown is byte-identical (SHA-256 verified), and all four extracted assets retain their hashes. The private manual and benchmark outputs are not committed.

CSS still uses the shared HTML style matcher. Existing Node document-object materialization, synchronous WASM calls, and the Rust example's second parse for --assets remain framework-level follow-ups outside this CHM-specific optimization.

Opening as a draft to discuss the reader/decompression strategy, private decoder adaptation, HTML adapter, TOC behavior, and memory tradeoff before treating this as ready for merge.

Summary by cubic

Adds native CHM (Compiled HTML Help) conversion. CHM files are now detected by their ITSF signature or .chm extension and convert through the shared document model across Rust, Node.js, Python, WebAssembly, and the CLI.

Reads ITSF/ITSP/PMGL directory structures from memory and decompresses the LZX content in one pass via a private lzxd adaptation, sharing decoded section ranges across entries so the byte budget is counted once; covers both ResetTable and SpanInfo-only stream framing.
Decodes legacy encodings (BOM, declared charset, or language-based fallback) with encoding_rs and adapts recovered HTML into the shared element tree with html5ever; HTML sibling mutation is constant-time via flat DOM ownership with weak sibling links.
Preserves the .hhc TOC hierarchy as a nested list, converts each topic once in TOC order, and appends unlisted topics in stable path order.
Retains referenced images through AssetSink; topic and TOC links use each file's selected encoding.
Adds a CHM fuzz target, fixture generator, binding tests for all runtimes, and reproducible release time/RSS benchmarks.

Known boundaries

Binary-only TOCs are not parsed; missing or binary TOCs fall back to deterministic path order.
Decoding the full solid section trades resident memory for bounded decompression work.