fix(app): drain offline-sync history oldest-first and honor 5xx Retry-After

Merged
GI
4d ago
+313-49

fix(app): drain offline-sync history oldest-first and honor 5xx Retry-After

What

Offline Sync could leave the oldest pending audio untried for weeks — observed on iOS 1.0.577: 395 pending phone WALs (3h27m of audio), the oldest 20.6 days — while every drain spent its progress on the newest recordings first, and unscoped 503s from /v2/sync-local-files failed batch after batch without any pause.

Two app-side changes:

  1. Deadline-ordered drain. nextSyncUploadBatch keeps live capture first (newest-first — the only audio that can still claim its conversation's capture manifest) and orders everything older oldest-first. The server permanently refuses audio older than SYNC_BACKFILL_MAX_AGE_SECONDS (30 days) with 422 backfill_lookback_exceeded, which retires the recording terminal (outsideRecoveryWindow). A drain that is interrupted — rate-limit break, cancelled run, app backgrounding — truncates from the tail, so ordering history newest-first starved exactly the recordings with the least recovery time left; oldest-first makes every interruption cost the audio with the most slack instead. Conversation/session/location grouping and batch budgets are unchanged.
  2. Paced 5xx handling. SyncUploadHttpException now carries the server's Retry-After when a 5xx provides one, and SyncUploadGate installs the in-memory backendBusy cooldown for it — one refusal stops the drain instead of failing every remaining batch against the same condition (the backend's sync_dispatch_unavailable and ledger-fence 503s always carry Retry-After: 30–60). Unscoped 5xx (platform shedding under load) get a growing inter-batch pause (2s doubling, capped at 30s) instead of immediately re-uploading the next full multipart body.

Production evidence (2026-10-26, times UTC)

  • Phone drain 03:24–03:30Z: SyncUploadHttpException(status=503) at batches 2/4/6/15/26 (terminal=false); wal_stale_missing pending_wal_count=395 pending_audio_seconds=12460 oldest_pending_age_seconds=1777438.
  • backend-sync Cloud Run request logs, POST /v2/sync-local-files 02:00–06:00Z: 10× 503; in the 03:25–03:30 slice 5× 503 vs 97× 202 and 154× 400. Two 503 families:
    • Platform 503 (209-byte body, zero application logs, latency 3–9.7s ≈ full body upload): backend-sync is pinned at its max-instance ceiling 24/7 (80 until 2026-10-24 08:00Z, 50 since); the request proxy sheds these after the body arrives. Fleet-wide on this route only: 463 on Oct-20, 78 on Oct-25, 10 through 05:30Z on Oct-26.
    • Application 503 sync_dispatch_unavailable (232-byte, sub-second, Retry-After: 30): DestructiveOperationInProgress from the legal-holds owner_storage_write_gate raised during _stage_files_to_gcs staging. Chronic: Oct-19 ≥1,892, Oct-20 341, Oct-21 ≥1,959 (both query-capped at 2,000), Oct-22 328, Oct-23 93, Oct-24 19, Oct-25 65, Oct-26 (partial) 8.
  • No loss from app retention: the admission count cap (720 vs retained 5,314, blocked_count=0) is warning-only — pending copies are never evicted and admission is never refused for count; autoRemoveSyncedCopies deletes only synced copies with a known syncedAt. The real cliff is the server's 30-day recovery window: the Oct 5 backlog starts being refused permanently around Nov 4, which is why the oldest files must drain first.

Product invariants affected

INV-CAP-1: neither change discards or evicts pending audio. Reordering only changes which durably-retained WAL uploads first, and the new 5xx pauses stop transmission while every pending file stays retained on disk — the invariant's allowed pause-while-retaining case; the 720-count cap and auto-remove policies are untouched.

Client-compat

Additive only: SyncUploadHttpException gains an optional named field (retryAfterSeconds); no wire-format or request changes.

Testing

  • flutter test test/unit/local_wal_sync_test.dart test/unit/wal_upload_resilience_test.dart test/unit/sync_upload_gate_test.dart test/services/wals/mobile_standard_batch_test.dart — all passing (235 tests). The batch-ordering contract is updated deliberately: historical batches now assert oldest-first; live-capture priority, conversation grouping, and budgets are unchanged.
  • New coverage: the gate installs the backendBusy cooldown on a 503 with Retry-After and refuses the next upload client-side; a drain stops after one refused batch with no retry-budget spend; an unscoped 503 drain keeps attempting every batch with a [2,4,8,16]s pause sequence; syncServerUnavailablePauseSeconds caps at 30s; mixed live/historical ordering.
  • Full app hermetic suite: bash test.sh — 7,504 passed, 9 skipped, 0 failed.
  • Backend untouched: BACKEND_UNIT_TEST_FILE_LIST=<empty> bash test.sh selects no backend unit tests.

Failure-Class: new

Line-Count-Exception: app/lib/services/wals/local_wal_sync.dart | 2611 -> 2636 | unscoped-5xx inter-batch pause needs the drain-loop streak state and a testable sleep seam beside the existing retry metadata in this file

UI-Evidence: none -- service-layer sync ordering and retry pacing only; no UI files changed

🤖 Generated with Claude Code