fix(backend): let desktop clients replay a Gemini provider outage

Merged
KO
2mo ago
+91-0

Bug

GET/POST through the desktop Gemini proxy fails the user's call outright whenever Google's endpoint returns a transient availability error.

Prod evidence — desktop-backend, 2026-08-16 → 2026-08-19, jsonPayload.event="desktop_gemini_proxy_terminal", outcome="provider_unavailable": 186 terminal events, every one at attempt: 1.

upstreamntop model / action
50382gemini-embedding-001:batchEmbedContents (80)
50219gemini-2.5-flash:generateContent (vertex_ai)
50015gemini-2.5-flash:generateContent

Steady across the window (8 in the last hour sampled). The 80 batchEmbedContents failures are the Rewind index path — each one is screen text that never gets embedded.

Root cause

_provider_error in backend/routers/desktop_proxy.py mapped every upstream >= 500 to a single terminal shape: proxy status 502, X-Omi-Retryable: false, no Retry-After.

An upstream 502/503 delivered nothing, so the request is safe to re-issue — and this proxy deliberately delegates the re-issue to the caller rather than retrying in-band. Both desktop clients read that decision off the wire, and the collapsed 502 revoked replay on both at once:

  • macOS — GeminiClient.httpError parses X-Omi-Retryable into apiError(_, retryable:), and shouldAutoRetry returns retryable == true. false ⇒ the retry loop rethrows on the first attempt (GeminiClient.swift:254, :423, :623).
  • Windows — renderer/src/lib/geminiClient.ts:62 and main/rewind/embeddingClient.ts:70 retry on res.status === 429 || res.status === 503 only. A 502 takes the throw branch immediately.

So the proxy converted a Google capacity blip into a hard user-visible failure, on both platforms, with the clients' own bounded retry (2 attempts, 1s/2s on macOS, 400/800ms on Windows) never firing.

Fix

_provider_error now classifies upstream 502/503 as what they are — an availability fault, not a verdict on the request:

  • proxy status 503 (so the Windows clients' existing 429 || 503 branch fires)
  • X-Omi-Retryable: true (so macOS shouldAutoRetry fires)
  • Retry-After: 1 — a floor only; both clients already back off on their own

Upstream 500 keeps the terminal 502. INTERNAL can be a verdict on the request body (oversized context), not a capacity signal, so it does not invite a replay of the same bytes.

No in-band retry is added: the proxy's replay-authorization contract is the existing seam, and _provider_error is shared by the streaming and non-streaming paths, so this is one change for both.

What I tested

backend/.venv/bin/python -m pytest backend/tests/unit/test_desktop_proxy.py

Three new tests, one of them driving the whole non-stream route (_proxy → real _provider_error → the headers a client actually receives) with only the httpx client, metering and routing stubbed:

  • test_provider_availability_fault_authorizes_client_replay[502|503]
  • test_proxy_reports_upstream_unavailable_as_retryable — real route, asserts the upstream body is not echoed
  • test_provider_internal_error_stays_terminal — pins the 500 boundary

Control on clean origin/main (git stash of the router change only): all three fail with assert 502 == 503. With the fix: pass.

  • test_desktop_proxy.py — 91 passed (163s)
  • test_desktop_core.py, test_desktop_backend_cors.py, test_desktop_rest_inventory.py, test_desktop_surface_parity.py, test_desktop_llm_stub.py — 38 passed

Failure-Class: FC-recoverable-provider-failure-terminal

🤖 automated by hourly watchdog — tested and merged