fix(backups): secure pCloud setup and durable conversation exports

Merged
AR
2d ago
+4685-119

Summary

Refs #20213. Builds on #20498 while preserving its contributor commits; do not merge the unsafe foundation first.

This PR completes the account/consent/durable-delivery boundary rather than adding another UID-only observer or uncorrelated audio buffer. Maintainer authorization now covers merging verified source and the repository's normal automatic development-deployment/app-release pipelines. No manual deployment, pCloud credential provisioning, provider activation, remote-data mutation or issue closure is requested.

Changes

  • Opt-in per-app backend transport: account-bound one-use setup capabilities, exact origin checks, scoped completion reads, signed finalized delivery, unchanged idempotency/SSRF/redaction for existing apps.
  • Secure cookie sessions, CSRF-protected POST settings/disconnect, explicit transcript/audio opt-in, authoritative US/EU OAuth callback routing and truthful pending revocation.
  • Encrypted transactional jobs with per-file receipts, stable account/conversation paths, bounded retries, restart leases, expiry scrubbing, key rotation and account/consent/generation fences.
  • Saved audio uses the canonical Omi playback path, not UID-only live buffers. Current app/account/conversation access is rechecked before provider writes. Missing/unready audio is visible, not reported as uploaded.
  • Real FastAPI/provider/store tests are registered in both deterministic manifest lanes; no production framework/model shims or public/default encryption key.

Architecture and rollout constraints

The standalone service requires one replica on an explicitly configured persistent volume. It does not silently fall back from Redis to JSON or deploy onto ephemeral Cloud Run disk. No existing Redis/file data is read, deleted or automatically migrated. Any existing installation must be inventoried and migrated explicitly before rollout.

The shared plugin SDK remains models-only. Transport and job architecture/checklist are documented in backend/docs/plugin-transport.md and plugins/omi-pcloud-app/README.md; capture/playback ownership is recorded in the pipeline guide. User-facing setup/status remains the existing mobile/desktop integration flow.

OMI_PLUGIN_TRANSPORT_JSON and the matching standalone configuration are intentionally unprovisioned. Reviewed runtime secret bindings on every finalization host, a pCloud developer app/test account, minimal provider permissions/callbacks, persistent-volume backup/restore, real connected US/EU export/retry/revocation and separately authorized rollout remain release gates. Live provider configuration has not been verified; local fixtures are not provider credentials. Code readiness and activation are separate. The new registered pCloud transport stays inactive until provisioned; activation remains blocked on the release gates above. Existing-client fresh-grant and credential-isolation fixes do affect later app builds, so the whole PR is not described as inactive. Keep #20213 open until they are verified.

Line-Count-Exception: backend/utils/apps.py | 1716 -> 1731 | Re-enable validation owns the probe list, so the registered-transport branch (trusted-origin check plus non-mutating health probe instead of an unsigned delivery) belongs inside validate_app_endpoints_for_reenable rather than a parallel validator.

UI evidence

UI-Evidence: none

Both app-home and setup-step actions fetch fresh launch grants. Flutter app_detail.dart, macOS AppsPage.swift, Windows AppDetailSheet.tsx and web AppDetail.tsx retain the existing layout. Web uses a same-styled button, reserves a tab synchronously, then navigates it to the fresh URL. A failed refresh closes the reserved tab and shows a retryable inline error. Local browser evidence below uses synthetic loopback grants, not a connected provider.

Review follow-up

  • Launch grants on every client: macOS, Windows and web reused the grant loaded with the sheet, so a second Open (or one after the grant expired) failed. Each now refetches app details right before opening, as Flutter already did.
  • Folder changes for unfinished uploads: a queued or retrying job kept the folder captured at enqueue. It now adopts the current destination before uploading and restarts its receipts there; a folder change mid-upload fails the job with a retry prompt instead of splitting it across folders.
  • Completed receipts after expiry: expiry scrubbed completed and canceled jobs too, wiping their per-file receipts. It now scrubs only unfinished payloads; rows still go after the 30-day purge.
  • Failed backups beyond 30 entries: the list was the newest 30 rows, so older failed jobs lost their retry. It now returns the newest 30 plus every failed or pending job (bounded by the existing 100 unfinished-job cap).
  • Each item has a test that failed before the fix: three plugin tests, plus a Windows and a web test that open twice and expect a new grant each time.

Second review follow-up (2026-10-23)

  • Disconnect retires pending OAuth attempts. Previously, with attempts A and B pending, completing A, disconnecting, then completing B restored an active connection. Disconnect now rotates a durable per-account authorization epoch (also when no connection exists). Connect stamps the epoch on the OAuth state; the callback rechecks it under the account lock before the token exchange (a retired code is never exchanged) and again after it (a token issued to a retired attempt is revoked, not stored). If pCloud does not confirm that revocation, the page says so and points to removing the app in pCloud; a newer connection is never overwritten.
  • Windows/Linux first-time Install uses fresh grants. Apps.tsx beginSetupFlow built setup and completion URLs from the undecorated catalog item. It now fetches this account's app details first, keeps signed query bytes via urlWithUidParam, and launches, polls or enables nothing after a failed read, an account change or an unmount. The Install button stays busy during the read, so a repeated attempt gets a new grant.
  • Regressions that failed before the fix: four plugin route tests (two pending attempts then Disconnect; Disconnect with no connection; Disconnect during a paused token exchange, with revocation confirmed and unconfirmed) and six Windows page-level Install tests. Guard tests also cover a newer connection surviving a stale exchange, a new Connect after Disconnect, another account staying connected, and canceled jobs staying canceled.
  • Evidence on 137509030a: plugin suite 73 passed (real FastAPI routes and SQLite, synthetic provider I/O, network denied); Windows Apps/AppDetailSheet/appInstall 61 passed, typecheck clean; isolated loopback service smoke (boot and restart) passed. No live pCloud OAuth: the developer console is signed in, but app creation awaits pCloud approval, so client credentials and a real token exchange remain unverified.

Earlier verification (before the 2026-10-23 update)

  • bash scripts/run-plugin-pcloud-tests.sh: 67 real installed-service/provider/storage/signing contract tests passed with live provider requests blocked, including completed/partial receipt preservation on disconnect.
  • Selected backend runner: 132 tests passed in ten isolated files at the unchanged local 0.30s CPU guard, covering app setup/install, real finalized fanout composition, saved-audio access and existing finalization/playback behavior.
  • Route-policy inventory: 27 tests passed at the same unchanged CPU guard; GET/HEAD declare scoped plugin authority without expanding the legacy missing-route baseline.
  • Pinned backend type check: 0 errors, 41,730 warnings; existing warning debt remains.
  • Actual local browser/HTTP path with synthetic provider IO: account-bound link consumption/cookie handoff, capability URL scrubbing, opt-in defaults, saved preferences, retry admission/completion and failed revocation checked. Desktop and 390px embedded production-page rendering inspected; no live pCloud account or native-client build was used.
  • Local PR preflight: 38 selected manifest checks passed. The failure-class recurrence/history audit explicitly skips this shallow checkout; full-history CI remains authoritative. Normal pre-push results are recorded separately below.
  • Normal pre-push gate passed without bypasses, including the same 60-test plugin suite, pinned backend type check, OpenAPI checks and three changed backend test files. Its broad 2,064-file selector was bounded by the repository's unchanged 40-file cap; this is not a claim that the complete backend/platform suite ran locally.

Local source verification is on bbace15c92, based on main c84b9138f4c46b76728465656e4653308c64a289. Main advanced ten non-overlapping commits during this work; final GitHub merge-tree checks remain a gate before any merge.

Expected consent behavior comes from #20213. Region routing, OAuth token behavior and revocation are defined by pCloud authorization, OAuth tokens and logout. Existing mock metadata/response conventions are not treated as authority over those contracts.

Failure-Class: FC-unsigned-disconnect-deletes-tokens

Line-Count-Exception: desktop/macos/Desktop/Sources/MainWindow/Pages/AppsPage.swift | 3748 -> 3759 | Fresh home and setup-step launches and page/account retirement stay in the existing detail-sheet owner; AppSetupURL owns fresh-read fencing and signed-query preservation.

Line-Count-Exception: backend/routers/apps.py | 2673 -> 2696 | Existing marketplace integration entry points need setup-capability decoration and one GET/HEAD handoff; signing, authorization and audio logic are extracted into focused utility modules, with route-composition tests.

Why not a new one-off guard: this PR provides the account-bound transport primitive and durable state owner, then exercises production HTTP/state transitions; the existing unsigned-disconnect class already records this trust-boundary failure. The new manifest entry runs that behavioral suite rather than scraping source for a preferred implementation.

UI-Evidence: none ā€” Python-rendered standalone setup is outside released-client UI paths; local synthetic screenshots and browser observations are recorded in this task, not advertised as deployed/customer proof.

2026-10-23 maintainer review and merge gates

Ready for code review at Aryan's request; the gates below block merge, not review. The receipt/folder/retry-history fixes from the prior review are present. This pass resolves the current-main manifest conflicts, refreshes both home and setup-step grants on every client, preserves signed query parameters on Windows, rejects stale/failed launch reads, and removes mobile vendor completion bearer forwarding, signed-URL logging and app-detail body logging. The existing catalog owner also rejects retired account/ABA detail reads. Existing legacy apps without a completion endpoint keep optional setup. Local evidence: 67 installed plugin/provider/store tests, 78 isolated backend tests, 25 Flutter launch/authorization/privacy tests, 46 Windows renderer/install tests, 10 native Swift URL/launch-boundary tests, final Flutter analyzer, and the complete web check (6 Bun smoke + 876 Vitest tests) pass. The full macOS debug build also passed in 619.16 seconds with two jobs; this is not macOS 14 runtime acceptance or the full desktop test suite. A real browser with the production AppDetail and synthetic loopback HTTP grants accepted distinct grants on repeated setup clicks, closed a reserved tab after a forced 503, and recovered on retry. These are synthetic provider boundaries, not live pCloud OAuth/US/EU exports or installed-client acceptance. The source fix is c43b217362; documentation-only follow-ups 7f34a3cdb2 and 9deac7421d repair the backend instruction budget and pin a removed historical page reference to its measured source snapshot. At that historical checkpoint those items remained open. The current code-readiness and separate activation decisions are recorded in the 2026-10-25 update below. The mobile vendor isolation intentionally overlaps #21827; retain that boundary when integrating the two PRs. No deployment or credentials provisioned.

Maintainer readiness and scope update (2026-10-25)

  • Final reviewed head: 922986c34d2f8e86595c48c48e7616d2c5fb7e46; no new blocking code defect identified in the current follow-up. Fresh local bash scripts/run-plugin-pcloud-tests.sh: 73 passed with synthetic provider IO and network denied. The changed transport implementation is unchanged from the previously reviewed head; prior valid browser, client-launch, storage, account/epoch and revocation regression evidence is reused, not presented as a fresh live-provider run.
  • Current-head CI is green: 46 successful checks, 15 appropriately skipped, none pending/failing before readiness. All six backend shards and aggregate, required Mobile Release Eligibility, backend hermetic gate, Windows renderer/build smoke, Swift contracts, Flutter checks, Android emulator acceptance and iOS compile passed. The earlier head's backend wall-budget failure is not a failure of this head.
  • Local metadata-only preflight: five selected checks passed; final diff check and merge-tree against current main are clean. No baseline, timing budget, branch protection or required check was bypassed. The earlier shallow local history skip is not represented as a full-history ratchet pass.
  • The maintainer explicitly accepted the repository's normal automatic development deployments and app release train accompanying this merge. This is not authorization to create/configure the pCloud developer app or provision its transport. Those remain separate operator work.
  • Keep OMI_PLUGIN_TRANSPORT_JSON and the standalone provider configuration unprovisioned. Before pCloud activation, verify approved company/provider ownership, minimal grants and exact callbacks, real US/EU export/retry/revocation, and persistent-volume backup/restore. Keep #20213 open; a source merge is not customer recovery or proof of live-provider operation.
  • This PR overlaps #21827 in mobile app setup and completion handling. Integrate the second PR against the merged first PR, retain vendor-bearer isolation/fresh grants and account/attempt ownership, regenerate API clients after schema merges, and rerun the affected tests.