feat(pdf): convert the readable pages when others need ocr
A PDF with one image-only page returns nothing at all, so the pages that do carry text are lost along with it. One image on page 3 of an 80-page contract costs the other 79 pages.
Ocr::Skip converts the pages that carry text instead of failing the document, and names the ones it left out: Options::default().ocr(Ocr::Skip) returning Conversion { markdown, pages_needing_ocr, page_count } in Rust, { ocr: 'skip' } in Node, ocr="skip" in Python, toMarkdownBytesWith(bytes, format, 'skip') in wasm, --ocr skip on the CLI. A document where no page yielded text still fails with NeedsOcr — there is nothing to hand back, and the error is what tells a caller the document needs OCR rather than better parsing, so --ocr hosted and exit code 3 keep working as the signal for a real scan.
The default is untouched, byte for byte: to_markdown / to_markdown_bytes delegate to the new *_with forms with Options::default(), and all 66 existing snapshots are unchanged. Options is #[non_exhaustive] with a consuming ocr() setter, so #102 and #146 have somewhere to land later. No dependency bump: pdf-inspector 1.14.2 already returns per-page Markdown with a per-page needs_ocr.
Nothing is written into the Markdown
#144's third suggestion was an in-text marker where a page was dropped. This reports the pages on the result instead and leaves the output as content alone, because that Markdown is shown to readers and fed to models — a notice injected into it is corruption downstream, and unlike a marker, a page list can be branched on. The objection behind #140 was that the loss was silent, and it is not silent if the caller is handed the page numbers and had to opt in to get them. A test pins the Markdown as free of any such notice, so this cannot drift.
On the CLI the same reasoning puts the report on stderr, which already carries every diagnostic, so stdout stays Markdown alone:
$ anydoc contract.pdf --ocr skip > contract.md
anydoc: converted 79 of 80 pages; page 3 needs OCR
Why per-page extraction, and where it costs
Whole-document extraction is all-or-nothing, not merely lossy: process_pdf_mem classifies a mixed document as ImageBased and returns markdown: None for the whole thing, even where individual pages carry perfectly good text. So the readable pages have to come from extract_pages_markdown_mem, and the per-page needs_ocr verdicts decide both what is converted and what is reported — the two cannot disagree.
That pass only runs once detection and the existing confirmation pass agree a page is genuinely unreadable, so a document that converts today pays nothing new. Measured over 30 runs, an 80-page file:
| default | ocr: 'skip' | |
|---|---|---|
| no page needs OCR | 4.50 ms | 4.53 ms |
| one page needs OCR | 5.62 ms (returns nothing) | 10.51 ms (returns 79 pages) |
The second pass doubles the cost on documents that need it, against returning nothing at all. Worth a look if that trade is not the one you want.
Ocr::Skip is also decided before conversion rather than after a failure, unlike the hosted recovery in #141: no throw, no second file read, no double parse.
Tests
handmade-partly-scanned.pdf is a new 5-page fixture, TEXT, IMAGE, TEXT, TEXT, IMAGE with per-page text, generated by the existing handmade_pdf() helper. The existing one-text-one-image fixture cannot show that pages after an image page survive, which is the reported failure. Its default-path snapshot is the unchanged ERROR: pages 2, 5 of 5 need OCR.
Five assertions in tests/snapshots.rs cover the recovered pages, the absence of notices in the output, the fully scanned document still erroring, text.pdf being byte-identical in both modes, and Options::default() matching the plain functions. Plus the Node, Python and wasm equivalents, and CLI coverage of both the stderr report and exit 3.
cargo fmt --check clean · cargo clippy --workspace --all-targets --all-features -- -D warnings exit 0 · cargo clippy -p anydoc-wasm --target wasm32-unknown-unknown exit 0 · 301 Rust tests, 24 Node, 13 Python, 9 wasm pass · cargo doc clean with -D warnings · Node overloads typecheck under --strict.
Unrelated, noticed on the way
A PDF that detection does not flag but which yields no text returns Unsupported, not NeedsOcr (the final match in src/formats/pdf.rs). Since the hosted fallback keys on code === 'needsOcr', such a document never reaches Parse even with --ocr hosted. Left alone here; happy to file it separately.
Fixes #144
Summary by cubic
Fixes #144: a PDF with scanned pages previously returned nothing; now ocr: 'skip' converts the readable pages and reports the ones it left out. The report is data on the result and stderr for the CLI, never a notice in the Markdown, and the default path is byte-for-byte unchanged.
New Features
- Adds
Ocr::SkiptoOptions, exposed asocr: 'skip'in Node and Python,--ocr skipon the CLI, andtoMarkdownBytesWith(bytes, format, 'skip')in wasm. - Returns a
Conversioncarryingmarkdown,pages_needing_ocr, andpage_count. - A document where no page yielded text still errors with
NeedsOcr, so--ocr hostedand exit code 3 still signal a real scan. - Per-page extraction adds a second pass, roughly doubling conversion time for documents that need it.