test(google-drive): add a mock Google Drive server and a connector benchmark

EV
1d ago
+5319-6

Description

Stack A (parallel Drive listing), PR 1 of 5. Plan: Indexing Scale-Out Plan, "Suggested order: test bed first".

This PR adds a test bed for the Google Drive connector. It changes no connector behavior.

  • Mock Google Drive server (backend/tests/integration/mock_services/mock_google_drive_server/). It serves the token, Drive v3, Admin SDK and Docs v1 calls. It copies Google's visibility rules, quotas and error shapes.
  • GOOGLE_API_ENDPOINT_OVERRIDE (testing only). It sends the Google connectors to another base URL. The env inventory baseline has the new name.
  • Benchmark (backend/scripts/benchmark_google_drive_connector.py). It runs the connector's checkpoint loop against the mock. It reports docs per minute, quota use, and each file that did not become a document.
  • Test (test_mock_drive_indexing.py). It indexes a small mock corpus and checks that each file becomes a document.

Taken from #17542

The mock server, the endpoint override with its resources.py, shared_constants.py and google_auth.py plumbing, the benchmark script and the test.

Not taken: the connector performance changes (rolling window, token reuse, retries, interleave, light PDF child). Those are for Stack B.

Changes from #17542

  • The benchmark does not set DRIVE_CONVERSION_MAX_IN_FLIGHT, which does not exist on main.
  • source_operations.py takes only the Docs API root URL change.

Left for later

  • The mock is not in docker-compose.mock-it-services.yml. The test starts it as a subprocess.
  • The mock already honors pageSize up to 1000 and modifiedTime range filters in q. The PR 4 tests depend on that.

How Has This Been Tested?

Run in a cloud session with no Docker. Postgres and Redis ran as local processes.

  • pytest backend/tests/external_dependency_unit/connectors/google_drive/test_mock_drive_indexing.py: pass.
  • python -m scripts.benchmark_google_drive_connector --start-mock with 240 files and the default mock latency: 240 of 240 documents, 0 failures, 0 duplicates, 249 docs/min. This is the baseline on main for the Stack B PRs.
  • ruff, ty, ods check-getattr, ods check-lazy-imports, the env drift gate and the feature-map check: pass.

Not run: the full pre-commit set (the session could not download the Go and Terraform hooks), the Docker image build of the mock, and CI.

Night 2 update

  • Fix: batch requests failed with ValueError: not enough values to unpack in BatchHttpRequest._header_to_id. The Google client folds a long Content-ID (for example a webViewLink key) over two lines. The mock echoed the folded value. The mock now unfolds it, as Google does (mock_gdrive/batch.py).
  • Test: test_mock_drive_batch.py reads files by webViewLink in one batch with the real client. It fails without the fix and passes with it, on this branch and on the stack tip.
  • Fix: the key from /_mock/service-account-key had no client_x509_cert_url, so the admin panel refused it ("The file is not a Google service account key"). The key now has every field of a real key. test_service_account_key_is_accepted fails without the fix.
  • Used live: a local isolated stack indexed the mock corpus (770, 2850 and 4450 files) through the admin API with GOOGLE_API_ENDPOINT_OVERRIDE.
  • The stack above this PR was rebased with no other change.

Update (2026-11-13): corpus shapes from the live Workspace

  • New corpus settings, all off by default (the default corpus does not change):
    • bulk_uploads: bursts of files with one identical whole-second modifiedTime, a set time apart, in a My Drive, a shared drive or an external account.
    • external_shares: files owned by external accounts (default domain gmail.com) and shared with N domain users. The token endpoint refuses to impersonate external accounts, as Google does.
    • folder_trees: a folder tree with a set depth and branching, plus one large deep folder.
  • New control endpoints for expected file sets: GET /_mock/corpus/folders/{id}/content-ids and GET /_mock/corpus/users/{email}/content-ids?scope=….
  • OAuth against the mock: get_google_oauth_creds sets the token URI to GOOGLE_API_TOKEN_URI before a refresh, only when GOOGLE_API_ENDPOINT_OVERRIDE is set. Production does not change.
  • README: a "Verified against live Drive (2026-11-12/13)" list and a "Known differences from live Drive" list.
  • Tests: test_mock_drive_corpus_shapes.py (8 tests).

Docs

  • Customer-facing change: I updated documentation/ (text and screenshots).
  • No docs change needed (internal, tests, CI, or multi-tenant cloud only).

Credits

Dane (Danelegend) wrote the mock server, the override and the benchmark in #17542.

Additional Options

  • [Optional] Please cherry-pick this PR to the latest release version.

https://claude.ai/code/session_01SK7J8QCWWmwbihV7zceSKD

[!NOTE]

Add mock Google Drive server and Google Drive connector benchmark

  • Adds a full mock Google Drive service: seeded corpus generation, Drive/Docs/Admin SDK API handlers, OAuth and service-account auth, quota and rate-limit tracking, latency and fault simulation, batch support, and content rendering to text, PDF, DOCX, PPTX, and other formats (api.py, corpus.py)
  • Adds a benchmark script that runs the Google Drive connector against the mock server and reports document counts, throughput, CPU, quota usage, and failures (benchmark_google_drive_connector.py)
  • Adds a GOOGLE_API_ENDPOINT_OVERRIDE app setting. When set, Google clients are built from bundled discovery documents with the replaced root URL, token refresh and Docs API calls use the override (resources.py, app_configs.py)
  • Adds integration tests for batch file lookup, service-account validation, and a full crawl that indexes every mock file
  • Risk: get_google_creds now points both OAuth and service-account refresh at GOOGLE_API_TOKEN_URI, and build_google_service raises ValueError for services without a bundled discovery document when the override is set — check backend/onyx/connectors/google_utils/resources.py

Macroscope summarized 6ba1281.

Summary by CodeRabbit

  • New Features
    • Google API, authentication, and Docs requests can be routed through a configurable endpoint.
    • Added a mock Google Drive service with configurable sample content, permissions, latency, quota limits, and API behavior.
  • Developer Tools
    • Added a benchmark for measuring Google Drive connector throughput, coverage, failures, and request statistics.
    • Added checks for batch requests, indexing coverage, and mock service behavior.
  • Documentation
    • Added setup and configuration guidance for the mock service and benchmark.

Update (night 3)

Second review round, f557007dca: GOOGLE_API_ENDPOINT_OVERRIDE accepts plain http only for localhost, 127.0.0.1 and ::1; eight mock server fixes (domain corpus, supportsAllDrives, inherited roles in permission queries, quota reset on a model change, UTC and millisecond times, JSON config answer, no infinite latency). Tests: 29 Drive mock tests and 9 override cases pass. 10 bot threads answered and resolved.

Generated by Claude Code