fix(sharepoint): back off on Graph throttling instead of failing

Merged
EV
1d ago
+259-16

Description

Evidence. The daily "Connector Tests" runs on main fail often. About 18 of 35 sampled red runs (Oct 6–19) end with RuntimeError: Failed to load documents ... Graph 429 activityLimitReached: The request has been throttled. The main test is test_sharepoint_connector_all_sites__docs_only. hierarchy_node_permissions and perm_sync_checks_live fail the same way. The run log shows the 429 on the drive listing: Failed to fetch drives for site '.../sites/Permisisonsync': ('activityLimitReached', ..., '429 Client Error ... /v1.0/sites/danswerai.sharepoint.com:/sites/Permisisonsync').

Cause. The RuntimeError comes from the test helper load_all_from_connector, which raises when the connector yields failures. The SharePoint gateway sends some reads through the office365 Graph SDK with a bare execute_query() and no retry:

  • the root site (tenant domain)
  • the site listing and its next pages
  • the site by URL (get_site_id)
  • the drive listing (list_drives)

A 429 on list_drives became a MicrosoftGraphError. Drive init caught it, yielded an entity failure and skipped the whole site.

The other paths already back off: the raw Graph GET (graph_api_get_json), the SharePoint REST and Entra reads (sleep_and_retry) and the downloads. They honour Retry-After through backoff_seconds and bound_server_wait.

Fix.

  • SharepointSourceOperations now sends these SDK reads through the shared sleep_and_retry. Each try builds the query on a new client, because a failed query stays queued on its client.
  • 429 and 503 wait for Retry-After. Without the header, they back off exponentially with jitter (5s, 10s, 20s, capped at 30s). Each read tries at most GRAPH_API_MAX_RETRIES + 1 times, the same limit as the raw GET.
  • The existing server-wait policy (bound_server_wait) still applies. It logs long Retry-After values and caps them on cloud.
  • Other refusals (403, 404 and so on) are not retried and behave as before.
  • Indexing, permission sync and the capability checks all use this gateway, so all three get the back-off.
  • Indexing has no dedicated rate-limit error type. The existing signal is MicrosoftGraphError with status 429, which OneDrive re-raises through fails_the_attempt. When Graph still throttles after the retries, the SharePoint site-level stages now re-raise it. Those stages are drive init, drive access, delta page, drive iteration and site pages. The attempt fails and the next attempt resumes from the checkpoint. Before, the site was skipped. MicrosoftGraphError.is_throttled names the check.
  • The capability checks already map a 429 to RateLimitedValidationError.

How Has This Been Tested?

  • New unit tests in test_sharepoint_graph_throttling.py use a fake SDK client:
    • a 429 with Retry-After waits that long, then succeeds
    • a 503 backs off, then succeeds
    • exhausted retries raise MicrosoftGraphError 429 after GRAPH_API_MAX_RETRIES + 1 tries
    • 403 and 404 are not retried
    • a throttled next page of the site listing is retried
  • A new test in test_load_checkpoint_resilience.py checks that a throttled drive init fails the attempt. It does not skip the site.
  • All unit tests in tests/unit/onyx/connectors/{sharepoint,microsoft_utils,onedrive} pass (596).
  • I did not run the live daily SharePoint test. The AWS SSO token had expired, and the local .vscode/.env has no SHAREPOINT_CLIENT_SECRET.

Additional Options

  • [Optional] Please cherry-pick this PR to the latest release version.
  • [Optional] Override Linear Check

🤖 Generated with Claude Code

Summary by cubic

Makes the SharePoint connector back off on Graph throttling instead of skipping sites and failing the daily connector tests.

The gateway's office365 SDK reads (root site, site listing, site by URL, drive listing) ran with no retry, so a Graph 429 skipped the whole site. These reads now go through the shared sleep_and_retry back-off.

  • 429 and 503 wait for Retry-After or back off with jitter, up to GRAPH_API_MAX_RETRIES times; other refusals (403, 404) are not retried.
  • Each retry builds the query on a new client, because a failed query stays queued on its client.
  • When Graph still throttles, the site-level stages re-raise the 429 or 503 so the attempt fails and the next one resumes from the checkpoint, instead of skipping the site.
  • Indexing, permission sync, and capability checks all get the back-off.
  • New unit tests cover Retry-After honoring, back-off with exhaustion, non-retried refusals, throttled listing pages, single-use retried queries, and checkpoint resilience.

Summary by CodeRabbit

  • Bug Fixes

    • Improved SharePoint reliability when Microsoft Graph requests are throttled or temporarily unavailable.
    • Automatically retries eligible requests using server-provided or fallback delays.
    • Preserves throttling errors after retry exhaustion so checkpoint-based processing can retry the attempt instead of silently skipping data.
    • Non-retryable access and not-found errors continue to fail without retries.

  • Tests

    • Added coverage for throttling, retry delays, pagination, retry exhaustion, and checkpoint recovery.