feat(connectors): add a checkpointed slim listing and one metadata check for Drive

EV
1d ago
+1000-349

Description

Stack A (Drive scale-out), PR A3. Two parts: a checkpointed slim listing, and one metadata check that decides from a Drive file's metadata whether indexing would skip it.

Checkpointed slim listing

  • CheckpointedSlimConnector[CT] (connectors/interfaces.py): optional, with the contract of CheckpointedConnector. One call takes a checkpoint and a time window, yields slim document stubs, hierarchy nodes and failures, and returns the next checkpoint.
  • SlimDocumentStub (connectors/models.py): extends SlimDocument with the modified time, mime type, size and a typed ListingPayload (DriveListingPayload for Drive). The payload keeps its subtype fields through JSON (as_type). Pruning and permission sync read only the SlimDocument fields.
  • Drive implements it with retrieve_slim_docs_from_checkpoint. A retrieval error on that path is a ConnectorFailure.

One metadata check for Drive

  • drive_file_skip_reason(file, ...) (doc_conversion.py) returns why indexing would skip a file, or None: shortcut or folder, link-only file when exclude_domain_link_only is on, size over the threshold, image when images are off, a tabular file with no raw-file callback, or an unsupported type. For types that are not native Google types it calls is_indexable_file_type from file_types.py (added in #17820).
  • One mime handler table. Conversion dispatches through it, and the metadata check reads it, so a type that conversion reads cannot be missing from the check.
  • One post-download check (_post_download_skip_reason) for outcomes only the content shows: export over the size limit, Forms API off, no text, nothing left after the text cap. A failed download stays a failure (#17555).
  • Callers: conversion (first step), the stub listing, reindex, pruning (retrieve_all_slim_docs) and permission sync all use the same check. The Drive permission sync connector gets the indexing image policy.

Release note: pruning now applies the indexing filters. The first prune after the upgrade can remove documents of files that the current settings exclude (for example, files over the size threshold, or images when images are off).

Decisions

  • Pruning applies the indexing filters: it deletes any document an indexing run would not index now.
  • SKIP_SPREADSHEET_CONTENT drops nothing: conversion still stages the raw spreadsheet file.
  • Listing paths pass has_raw_file_callback=True: docfetching always sets one. Conversion passes its real value.
  • SLIM_FILE_FIELDS asks for size, which the stub and the size check need.

How Has This Been Tested?

  • Unit: test_drive_file_skip_reason.py (one case for each handled mime type, unsupported types, size, images, tabular without callback, shortcuts and folders, link-only).
  • External dependency unit against the mock Drive server (PR A1): test_checkpointed_slim_listing.py (resume with a JSON round trip each call; payload through JSON; size threshold) and test_pruning_omits_files_that_indexing_skips.
  • #17555's test_drive_download_failure.py passes on the new handler table.
  • At this head: Drive unit + file_processing tests, 509 passed.

Docs

  • Customer-facing change: I updated documentation/ for the page's reader and task, at the level of detail of the whole page.
  • No docs change needed: the existing docs stay correct (internal, tests, CI, multi-tenant cloud only, or the UI explains it).

Additional Options

  • [Optional] Please cherry-pick this PR to the latest release version.

🤖 Generated with Claude Code

Summary by cubic

Adds a CheckpointedSlimConnector interface and a SlimDocumentStub model so a slim listing can stop and resume without losing progress. Google Drive implements it via retrieve_slim_docs_from_checkpoint; no worker calls it yet. All Drive listings now filter by one metadata indexability check, so pruning and permission sync drop the files indexing skips.

  • A listing call yields slim document stubs, hierarchy nodes, and failures, then returns the next checkpoint.
  • Stubs add the modified time, mime type, size, and a listing payload; pruning and permission sync still read only the SlimDocument fields. The payload keeps its subtype fields through JSON and reads them back via as_type.
  • drive_file_skip_reason now gates indexing, the stub listing, reindex, pruning, and permission sync, so pruning removes documents of files indexing skips: shortcuts and folders, link-only files when excluded, oversized files, images when off, and unsupported types. A Google Form stays indexable since the Forms API reads it.
  • Conversion dispatches through one mime-type handler table that the check reads, so the check accepts each type conversion reads. Outcomes only the content shows (export over the size cap, Forms API off, no text) are decided during conversion.
  • Retrieval errors become ConnectorFailures on the stub path instead of raising.
  • The Drive permission sync connector now takes the indexing image policy, so its listing admits the image files indexing admits.
  • SLIM_FILE_FIELDS now asks for size; get_file_ext moved to file_types.py and is_indexable_file_type was added there.

[!NOTE]

Add CheckpointedSlimConnector interface and Google Drive slim listing with SlimDocumentStub

  • Adds a CheckpointedSlimConnector abstract interface in interfaces.py with checkpoint retrieval, dummy checkpoint creation, and JSON checkpoint validation
  • Implements a checkpointed slim-listing entrypoint on GoogleDriveConnector in connector.py. In indexing mode (stub_failures provided), retrieval errors are collected as ConnectorFailure values instead of raised, unindexable files are skipped, and stubs are built
  • Adds drive_file_can_be_indexed (filters shortcuts, folders, oversized files, and disallowed MIME types/extensions) and build_slim_document_stub in doc_conversion.py
  • Adds ListingPayload, SlimDocumentStub, and Drive-specific DriveListingPayload models in models.py and models.py; SLIM_FILE_FIELDS now also requests file size
  • Covers the listing with integration tests against a mock Drive server in test_checkpointed_slim_listing.py, including resume completeness, JSON round-trip, size threshold, and indexability cases
  • Behavioral Change: outside indexing mode, _extract_slim_docs_from_google_drive still raises retrieval errors and builds ordinary slim documents; no worker calls the new interface yet

Macroscope summarized 96abbe9.

Summary by CodeRabbit

  • New Features
    • Google Drive indexing can now resume file listings across multiple batches, with listing metadata preserved for subsequent processing.
  • Bug Fixes
    • Google Drive files are filtered more consistently by type, size, image settings, and content availability across indexing and permission updates.
    • Files that cannot be extracted in supported cases are skipped instead of interrupting processing; transient parsing errors can still surface.
  • Documentation
    • Updated connector documentation to describe resumable listings and Google Drive file eligibility.