perf(google-drive): convert files in a rolling window while listing goes on

Merged
EV
1d ago
+437-248

Description

Stack B (quick wins in the Drive connector), B1. Plan section 1.1, "Rolling window".

Base. The base is main (since 2026-11-12). The benchmark numbers below came from runs on top of #17548, which has the benchmark script. This PR no longer edits that script: when both PRs merge, --conversion-workers must also set DRIVE_CONVERSION_MAX_IN_FLIGHT (a one-line follow-up).

Before: the connector converted files in groups of 50 on 8 threads and waited for the whole group. A slow file held up the other workers. Nothing converted while the listing and the folder walk ran.

Now:

  • BoundedParallelRunner (onyx/utils/threadpool_concurrency.py) runs calls on a thread pool with a bound on the calls in flight, and hands back results as they finish. It copies the caller's context variables to each call.
  • The connector submits a file when its parent node is yielded, and goes on listing. Up to 32 conversions run. At most 64 are submitted and not yet yielded.
  • GOOGLE_DRIVE_CONVERSION_WORKERS sets the worker count. The default is 32.

Decisions

  • Default of 32. #17542 measured that past 32 workers one process uses about one core for parsing, so more workers do not help. On a small docfetching pod this is more threads and more requests in flight than today (8). The env var lowers it.
  • Memory. Up to 64 results wait at one time, against 50 before. Each is at most CONNECTOR_MAX_EXTRACTED_TEXT_CHARS.
  • Not taken from #17542: the interleave by user (a separate Stack B PR) and the parallel folder prefetch. Plan section 1.0b replaces the folder walk, so the prefetch is left out.
  • Conflict with Stack A. This PR and #17550 both move the retrieval-failure code into a _retrieval_failure helper with the same body. The second to merge has a small conflict in connector.py.

How Has This Been Tested?

  • Unit tests from #17542: test_threadpool_concurrency.py and test_threadpool_contextvars.py (the runner), and test_drive_conversion_batching.py rewritten for the rolling window (bounded concurrency, incremental yield, nodes before their documents, deferred files, skipped conversions). 264 pass with the other Drive unit tests.
  • test_mock_drive_indexing.py against the mock: pass.
  • Mock benchmark, default mock latency, 4 cores:
CorpusDocs per minuteMost requests in flight
240 files, mixed types#17548 (main)2498
this PR36733
520 files, more Google Docs#17548 (main)3768
this PR57233

All runs: every file became a document, 0 failures, 0 duplicates. PDF child processes still use about 100 CPU-seconds in each run; #17553 removes that.

  • ruff, ty, ods check-getattr, ods check-lazy-imports, the env drift gate, the feature-map check and format_docs.py --check: pass.

Not run: a live Workspace, the full pre-commit set and CI.

Docs

  • Customer-facing change: I updated documentation/ (text and screenshots).
  • No docs change needed (internal, tests, CI, or multi-tenant cloud only).

documentation/deployment/configuration/configuration.mdx lists GOOGLE_DRIVE_CONVERSION_WORKERS.

Credits

Dane (Danelegend) wrote the runner and the conversion change in #17542.

Additional Options

  • [Optional] Please cherry-pick this PR to the latest release version.

https://claude.ai/code/session_01SK7J8QCWWmwbihV7zceSKD

Generated by Claude Code

Summary by cubic

Speeds up Drive indexing by converting files in a rolling window while the listing continues instead of waiting on fixed groups. Before, files converted in groups of 50 on 8 threads and a slow file held up the rest; now up to 32 conversions run concurrently and results are returned as they finish.

  • Adds BoundedParallelRunner in backend/onyx/utils/threadpool_concurrency.py, which caps in-flight calls, returns results as they complete, and carries the caller's contextvars.
  • The connector submits a file once its parent node is yielded and keeps listing; at most 64 conversions are pending at a time. Finished conversions are collected before skip and failure checks so filtered files don't hold results back.
  • New GOOGLE_DRIVE_CONVERSION_WORKERS env var sets the conversion worker count (default 32, falls back to 32 on an invalid value) and is documented. Reindex and slim-document builds keep using 8 workers via DRIVE_FETCH_WORKERS.
  • Mock benchmark: 367 vs 249 documents/minute on a 240-file corpus and 572 vs 376 on a 520-file corpus.

[!NOTE]

Convert Google Drive files concurrently in a rolling window during listing

  • Google Drive file conversion now runs on a shared thread pool while the listing loop continues, instead of converting sequential sub-batches after listing. Results are yielded as they complete, and all outstanding conversions are drained before the connector call returns.
  • Adds a generic BoundedParallelRunner in threadpool_concurrency.py that caps in-flight futures, copies the caller's context into each task, and propagates task exceptions through result collection.
  • Worker count is now configurable via the new GOOGLE_DRIVE_CONVERSION_WORKERS env var (default 32, minimum 1), replacing the previous fixed value of 8. The in-flight limit is twice the worker count.
  • Conversion results of None are now dropped instead of emitted.
  • Behavioral Change: the connector no longer converts in fixed parallel batches; tests in test_drive_conversion_batching.py were rewritten to verify bounded concurrency, incremental yields, and skipped-conversion dropping.

Macroscope summarized 1e9417a.

Summary by CodeRabbit

  • Performance
    • Google Drive indexing downloads and converts files concurrently, with a configurable worker limit.
    • Converted files are yielded as they become available while retrieval continues. Parent nodes are yielded before their dependent files.
    • Conversion work is bounded to limit the number of files being processed at once.
  • Documentation
    • Added details on Google Drive conversion behavior and the worker-count setting, which defaults to 32.