perf(file-processing): give the PDF text child process light imports

Draft
EV
1d ago
+82-50

Description

Stack B (quick wins in the Drive connector), B3. Plan section 1.1, "Light PDF child". This PR is independent: its base is main.

read_pdf_file extracts the text of each PDF in a fresh child process. The child imports the module that holds the function it runs. That module was extract_file_text, so each PDF paid for all of its imports.

This PR moves the PDFium extraction to onyx/file_processing/pdfium_text.py, which imports only psutil and pypdfium2. The function body does not change. extract_file_text imports the function and TEXT_SECTION_SEPARATOR from the new module.

Decisions

  • The function is now public (extract_pdf_text_pdfium), because another module calls it.
  • A new test starts a fresh interpreter, imports pdfium_text, and fails if extract_file_text, openpyxl or chardet is loaded. It keeps the module light.

How Has This Been Tested?

  • pytest backend/tests/unit/onyx/file_processing: 221 pass.
  • Start cost of one child, measured as python -c "import <module>": 2.33 CPU-seconds for extract_file_text, 0.04 for pdfium_text.
  • Mock benchmark (from #17548), 240 files, default mock latency, 4 cores:
Docs per minuteCPU in PDF subprocesses
main24998.2 s
this PR3983.8 s

Both runs: 240 of 240 documents, 0 failures, 0 duplicates.

  • ruff, ty, ods check-getattr, ods check-lazy-imports: pass.

Not run: the full pre-commit set and CI.

Docs

  • Customer-facing change: I updated documentation/ (text and screenshots).
  • No docs change needed (internal, tests, CI, or multi-tenant cloud only).

Credits

Dane (Danelegend) wrote this change in #17542.

Additional Options

  • [Optional] Please cherry-pick this PR to the latest release version.

Generated by Claude Code

Summary by cubic

Moves PDFium text extraction behind read_pdf_file into a new lightweight module, onyx/file_processing/pdfium_text.py, so each PDF's child process no longer imports heavy dependencies like openpyxl and chardet. Child-process start time drops from about 2.33 CPU-seconds to 0.04 CPU-seconds, with no change to extraction behavior.

  • Adds a test that runs a fresh interpreter and fails if the new module pulls in extract_file_text, openpyxl, or chardet.