feat(gmail): re-fetch failed documents by id (Resolver)

Merged
EV
3d ago
+810-9

Description

GmailConnector now implements Resolver.reindex. Targeted reindex can re-fetch failed Gmail threads by document id.

Document types

  • Covered: email threads. Gmail emits only this type. The document id is the Gmail thread id, the same id the crawl yields.
  • Not covered: none.

How reindex finds the mailbox
A Gmail thread id is unique only inside one mailbox. The id does not name the mailbox. The crawl reads every Workspace user's mailbox through service-account impersonation (or one mailbox with OAuth).

  1. Crawl failures now link to their mailbox: https://mail.google.com/mail/u/<email>/#all/<thread id>. Reindex reads the mailbox from this link and fetches there first. This changes only the ConnectorFailure link, not the document's section links.
  2. Ids without that link (admin-requested targets, older failures) are looked up in every mailbox the crawl reads, the primary admin's first. Reindex stops when all ids resolve.
  3. Each mailbox lookup uses batched threads.get calls (50 per batch). 403/404 means "not in this mailbox". Rate limits and 5xx errors are retried, and become per-thread failures if they persist. They never count as "not found".
  4. A mailbox that cannot be impersonated (suspended user) is skipped. If the primary admin's mailbox fails, the error is raised, because the credential is bad. Missing scopes raise PermissionError, as in the crawl.

Failures

  • An id that is not a hex thread id yields a ConnectorFailure with no API call.
  • An id found in no mailbox yields a ConnectorFailure.
  • Entity failures are skipped, as in Confluence.

Filter parity with the crawl
The crawl applies these filters. Reindex applies each one:

  • Mailbox scope: the crawl reads only the users from _get_all_user_emails. Reindex searches only those mailboxes. A mailbox link that names another user is ignored.
  • threads.list without includeSpamTrash: the crawl never lists a thread whose messages are all in Spam or Trash. Reindex reads labelIds and yields a ConnectorFailure for such a thread. It then removes the labels, so the document text matches the crawl's.
  • Mailbox not provisioned (400 failedPrecondition): the crawl skips the mailbox. Reindex treats it as "not in this mailbox".
  • Thread with no messages: the crawl yields nothing. Reindex yields a ConnectorFailure.
  • The time-range query is not an access filter, so reindex does not apply it.
  • The connector has no other config (only batch_size).

Permissions
The crawl always sets external_access to the mailbox owner's email (thread_to_document). Reindex uses the same function, so include_permissions changes nothing. Perm sync (ee/.../gmail/doc_sync.py) gives the same single-owner access.

How Has This Been Tested?

Unit: uv run pytest -v backend/tests/unit/onyx/connectors/gmail/ gives 10 passed. Daily: uv run --env-file .vscode/.env pytest -v backend/tests/daily/connectors/gmail/ gave 10 passed before the rebase onto main (8 resolver, 2 existing crawl tests).

  • New backend/tests/daily/connectors/gmail/test_gmail_resolver.py (service account, real test workspace): one thread from a non-admin mailbox, all four threads across four mailboxes, mailbox link hint, bogus but well-formed id, unparseable id, empty input, entity-only input, and include_permissions=True with external_access checked per owner.
  • New unit tests in backend/tests/unit/onyx/connectors/gmail/test_connector.py: a thread with all messages in Spam/Trash yields a failure; a listed thread yields a document without label text; OAuth path with a mocked Gmail client: the Admin SDK returns 403, so reindex reads only the admin mailbox, and a thread from another mailbox yields a failure.
  • OAuth is not tested live. CI keeps no Gmail OAuth credential, because a token that needs refreshing can make CI fail from time to time. The unit test above covers the OAuth path.

Additional Options

  • [Optional] Please cherry-pick this PR to the latest release version.
  • [Optional] Override Linear Check

🤖 Generated with Claude Code

Summary by cubic

Implements Resolver.reindex on GmailConnector so documents that failed during a crawl can be re-fetched by Gmail thread id.

  • Gmail thread ids don't name their mailbox: reindex reads the mailbox from the crawl failure link when available, otherwise searches every mailbox the crawl reads with batched threads.get calls (50 per batch), admin first, stopping when all ids resolve.
  • Rate limits and 5xx errors are retried at the request and whole-batch level; persistent ones become per-thread failures and never count as "not found".
  • Reindex applies the crawl's filters: threads in Spam/Trash, empty threads, and threads in no crawled mailbox yield ConnectorFailures; invalid ids fail without an API call.

Summary by CodeRabbit

  • Improvements
    • Gmail reindexing retrieves threads in batches for faster, more reliable processing.
    • Temporary rate-limit and server errors are retried automatically.
    • Reindexing selects the appropriate mailbox and excludes Spam- and Trash-only threads to match crawl results.
    • Mailbox-specific links and permissions are preserved in failure details.
    • Invalid, inaccessible, unavailable, or unresolved threads now produce item-level failures without blocking other items.
    • OAuth reindexing is restricted to the administrator mailbox.