feat(pdf): export PDFs to DocLang .dclx archives
LU
1d ago+24743+24743+01234567890123456789012345678901234567890123456789-11-11-01234567890123456789
Add export to the DocLang 0.9 standard format for documents. It will help to have more exchange between Firecrawl and the Docling communities. Also, we get a nice viewer for parsed documents for free (https://doclang.ai/viewer/).
This PR is still initial work and limited to PDF documents. My agent had a blast building it, I did some reasonable review of the code and tests.
Summary
to_doclang/to_doclang_bytes: PDFs export to.dclx(OPC ZIP) archives containingdocument.xml, extracted image/page rasters underassets/andpages/, and the original PDF under/source.- Scanned pages work without OCR: image-only pages become
<picture>elements with location boxes —NeedsOcris no longer the outcome for this path. - Text PDFs keep structure: positioned runs, inline styles, headings, links, AcroForm fields (
<field_item>), drawn images, and page geometry are preserved where practically possible. - Lists: bullet/ordered markers + indentation →
<list>/<ldiv>/<marker>rather than flattened text. - Tables: tagged
/Tablestructure-tree elements are matched by MCID, and a ruled-table engine clusters drawn axis-aligned rules into grids (spans →lcel/ucel/xcel); both emit OTSL-token<table>blocks.DoclangHints::pages_with_tablesscopes the geometric detector. - Renderer lives under
src/render/doclang/— DocLang is an output format, not an input format. - Robustness: malformed operands never corrupt the CTM, paths paint under the CTM active at the stroke, clustered boundaries can't chain-merge, decoded pixels and archive/ZIP sizes are capped (
ResourceLimit), page content and Form XObjects decode once per page, and the document parses at most twice (upstream parse-dedup filed as firecrawl/pdf-inspector#662).
Test plan
-
cargo test— 315 unit + integration tests (lists, tagged/ruled tables, spans, headers, q/Q CTM, malformed operands, image caps, packaging limits,.dclxround-trip) -
cargo clippy -- -D warningsclean -
cargo fmt --checkclean - DocLang XML validated against the DocLang XSD in the integration tests
- Reviewer: convert a scanned PDF and a text+table PDF via
examples/convert.rs
Generated with Devin
Summary by cubic
Adds DocLang .dclx archive export for PDFs through new to_doclang / to_doclang_bytes APIs, now also exposed from Python. Scanned pages no longer end in NeedsOcr — image-only pages export as <picture> elements with extracted rasters, so this path works without OCR.
- Text PDFs preserve positioned runs, inline styles, headings, links, AcroForm fields, drawn images, metadata titles, and page geometry.
- Lists and tables are emitted as DocLang
<list>/<table>blocks via tagged/Tablestructure-tree elements and a ruled-line grid engine. - The original PDF goes under
/source; rasters pass through as JPEG/JP2 or wrap to PNG/TIFF underassets/andpages/. DoclangHintstoggles list/table detection and scopes the geometric table detector to specific pages; Pythonto_doclang/to_doclang_bytesaccept the samelists,tables, andpages_with_tablesoptions.- Export is cross-checked against Docling's own groundtruth exports, covering lists, tagged/ruled tables, RTL text, forms, and scanned pages; results round-trip through Docling's DocLang deserializer.
Robustness
- All PDF streams (page content, Form XObjects, images, palettes, masks) decode through a bounded Flate/LZW/ASCII85 decoder with strictly validated TIFF/PNG predictors, so decompression and pixel buffers can't exhaust memory.
- Malformed operands, oversized ruled grids, and unsupported color spaces degrade without corrupting the CTM or table layout; rejected fragments return their items to normal flow.
- 8-bit indexed images export correctly instead of panicking on a mask overflow, and non-BOM metadata titles decode as PDFDocEncoding.
- Documents parse at most twice per export, and page content and Form XObjects decode once per page.