feat(pdf): export PDFs to DocLang .dclx archives

LU
1d ago
+24743-11

Add export to the DocLang 0.9 standard format for documents. It will help to have more exchange between Firecrawl and the Docling communities. Also, we get a nice viewer for parsed documents for free (https://doclang.ai/viewer/).

This PR is still initial work and limited to PDF documents. My agent had a blast building it, I did some reasonable review of the code and tests.

Summary

  • to_doclang / to_doclang_bytes: PDFs export to .dclx (OPC ZIP) archives containing document.xml, extracted image/page rasters under assets/ and pages/, and the original PDF under /source.
  • Scanned pages work without OCR: image-only pages become <picture> elements with location boxes — NeedsOcr is no longer the outcome for this path.
  • Text PDFs keep structure: positioned runs, inline styles, headings, links, AcroForm fields (<field_item>), drawn images, and page geometry are preserved where practically possible.
  • Lists: bullet/ordered markers + indentation → <list>/<ldiv>/<marker> rather than flattened text.
  • Tables: tagged /Table structure-tree elements are matched by MCID, and a ruled-table engine clusters drawn axis-aligned rules into grids (spans → lcel/ucel/xcel); both emit OTSL-token <table> blocks. DoclangHints::pages_with_tables scopes the geometric detector.
  • Renderer lives under src/render/doclang/ — DocLang is an output format, not an input format.
  • Robustness: malformed operands never corrupt the CTM, paths paint under the CTM active at the stroke, clustered boundaries can't chain-merge, decoded pixels and archive/ZIP sizes are capped (ResourceLimit), page content and Form XObjects decode once per page, and the document parses at most twice (upstream parse-dedup filed as firecrawl/pdf-inspector#662).

Test plan

  • cargo test — 315 unit + integration tests (lists, tagged/ruled tables, spans, headers, q/Q CTM, malformed operands, image caps, packaging limits, .dclx round-trip)
  • cargo clippy -- -D warnings clean
  • cargo fmt --check clean
  • DocLang XML validated against the DocLang XSD in the integration tests
  • Reviewer: convert a scanned PDF and a text+table PDF via examples/convert.rs

Generated with Devin


Summary by cubic

Adds DocLang .dclx archive export for PDFs through new to_doclang / to_doclang_bytes APIs, now also exposed from Python. Scanned pages no longer end in NeedsOcr — image-only pages export as <picture> elements with extracted rasters, so this path works without OCR.

  • Text PDFs preserve positioned runs, inline styles, headings, links, AcroForm fields, drawn images, metadata titles, and page geometry.
  • Lists and tables are emitted as DocLang <list>/<table> blocks via tagged /Table structure-tree elements and a ruled-line grid engine.
  • The original PDF goes under /source; rasters pass through as JPEG/JP2 or wrap to PNG/TIFF under assets/ and pages/.
  • DoclangHints toggles list/table detection and scopes the geometric table detector to specific pages; Python to_doclang/to_doclang_bytes accept the same lists, tables, and pages_with_tables options.
  • Export is cross-checked against Docling's own groundtruth exports, covering lists, tagged/ruled tables, RTL text, forms, and scanned pages; results round-trip through Docling's DocLang deserializer.

Robustness

  • All PDF streams (page content, Form XObjects, images, palettes, masks) decode through a bounded Flate/LZW/ASCII85 decoder with strictly validated TIFF/PNG predictors, so decompression and pixel buffers can't exhaust memory.
  • Malformed operands, oversized ruled grids, and unsupported color spaces degrade without corrupting the CTM or table layout; rejected fragments return their items to normal flow.
  • 8-bit indexed images export correctly instead of panicking on a mask overflow, and non-BOM metadata titles decode as PDFDocEncoding.
  • Documents parse at most twice per export, and page content and Form XObjects decode once per page.