fix(desktop): refresh relay JWT on WS reconnect and stop retrying definitive 403s

Merged
KI
3mo ago
+231-22

Summary

  • Terminals and the host event bus no longer get permanently stuck "Disconnected" after the hourly relay JWT rotation: the desktop now refreshes the JWT on demand and re-signs the WS URL on every reconnect attempt.
  • A definitive access denial (preflight 403) now stops the terminal reconnect loop with a clear "no access" message, and drops the event bus to a 5-minute poll instead of hammering the relay every 1-30s.

Why / Context

PostHog terminal_connect_failed shows ~25 users/day failing terminal connects with preflight_status: 401 (1,354 events since telemetry landed on 07-09), and relay logs show matching clients retrying _whoowns + terminal WS with a dead token every ~2s. Root cause chain:

  • Relay JWTs expire after 1h (packages/auth/src/server.ts jwt plugin config; the relay even special-cases ERR_JWT_EXPIRED as an expected, common failure).
  • The renderer's cached JWT (auth-client.ts) only updated when some API response happened to carry a set-auth-jwt header.
  • Terminal reconnects reused transport.currentUrl verbatim — the token was baked in at render time — so any terminal older than ~1h that hit a reconnect (laptop sleep/wake, host-service restart, network blip) redialed with an expired token forever. Restarting the app "fixed" it.

Separately, relay logs showed ~99% of cross-region proxied /events upgrades failing 403 (per-user v2UsersHosts access vs the org-wide host list the desktop connects to), each denied client retrying every few seconds indefinitely.

How It Works

  • auth-client.ts: setJwt records the token's exp (lib/jwt-expiry.ts). New ensureFreshJwt() returns the cached token while it has >60s left, otherwise mints a fresh one from better-auth's /api/auth/token (deduped across concurrent callers; falls back to the stale token if the request fails). getJwt() kicks a background refresh when the cached token is near expiry so sync callers (event bus URL builder, tRPC headers) converge on the next attempt.
  • terminal-ws-transport.ts: every relay dial — including reconnects — awaits ensureFreshJwt() and re-signs the URL's token param before the _whoowns preflight and the WebSocket open. Local PSK terminals are untouched. A preflight 403 is now fatal: log the diagnosis, emit terminal_connect_failed, stop the loop (a manual reconnect still re-enters connect()).
  • eventBus.ts: a preflight 403 reschedules at a fixed 5-minute interval instead of the exponential 1-30s loop, so access granted later (host sharing) is still picked up without the retry storm.

Manual QA Checklist

  • Open a remote-host terminal, leave it idle >1h, force a reconnect (toggle wifi or sleep/wake) — terminal reattaches instead of looping on 401
  • Local (PSK) terminals still connect and reconnect normally
  • A host you have no access to shows "You don't have access to this host. Not retrying." in the terminal pane log instead of silent retries
  • Relay logs: no more per-second _whoowns+terminal 401 retries from a stale client after the fix

Testing

  • bun run typecheck (desktop + workspace-client via turbo)
  • bun run lint
  • bun test apps/desktop/src/renderer/lib packages/workspace-client — new jwt-expiry.test.ts passes (and fails when the impl is mutated); the 10 pre-existing font-test failures fail identically on a clean tree
  • Verified https://api.superset.sh/api/auth/token exists in prod (401 unauthenticated) and that better-auth 1.6.13 returns { token } from it, matching how ensureFreshJwt reads the response
  • Not tested end-to-end with a real expired-token reconnect (needs an hour-old session in a dev desktop); the QA checklist above covers it

Known Limitations

  • apps/web has its own TerminalConnection.ts with the same stale-token flaw Correction: web's getAuthToken() already TTL-caches and refetches per connect attempt, so web is not affected. Consolidating all three hand-rolled reconnect loops onto partysocket is tracked in SUPER-1469.
  • The product-level 403 question remains: the desktop connects to every online host in the org (Electric v2Hosts is org-scoped) while relay access is per-user (v2UsersHosts, creator-only by default). This PR makes clients behave sanely when denied; whether teammates should get access automatically needs a team decision.

Risks / Rollout

  • Risk: /api/auth/token gets called on token expiry from WS reconnect paths — deduped and low-volume (bounded by reconnect backoff), so no auth-endpoint storm (cf. the 07-08 self-DDoS incident; this reduces relay retry traffic overall).
  • Rollback: revert the commit; behavior returns to current prod (401 loops until app restart).
Open in Stage

Summary by cubic

Refreshes relay JWTs on every WebSocket reconnect and stops futile retries when access is definitively denied. Adds race guards so token refreshes and reconnects can’t clobber newer state.

  • Bug Fixes

    • JWT handling (desktop): track exp, use 60s leeway, and add ensureFreshJwt() to mint from better-auth /api/auth/token (deduped, generation-safe). getJwt() starts a background refresh near expiry and logs failures.
    • Terminal transport (desktop): re-sign the relay WS URL with a fresh token on every dial; treat preflight 403 as fatal with a clear “no access” message and emit telemetry; epoch-guard preflight callbacks to avoid duplicate sockets or terminating newer attempts. Local PSK terminals unchanged.
    • Event bus (@superset/workspace-client): on preflight 403, switch to a fixed 5-minute retry instead of exponential 1–30s.
    • Relay (apps/relay): _whoowns now runs cached access checks when this edge owns the tunnel and returns 403 on denial (500 on check error), making denials visible to clients.
    • Tests: add jwt-expiry utility and unit tests.

  • Refactors

    • Docs: add a design plan to consolidate reconnecting WS clients on partysocket (SUPER-1469).

Summary by CodeRabbit

  • New Features
    • Added expiry-aware JWT caching with background refresh to keep sessions valid.
    • Terminal reconnects on relay routes now request a fresh JWT before retrying.
    • Relay preflight now surfaces clearer access outcomes (including definitive denials).
  • Bug Fixes
    • Improved handling of missing or malformed JWT expiry data.
    • Stops auto-reconnect loops after explicit access denial (403).
  • Reliability
    • Uses a slower fixed reconnect delay when access is denied (403).
  • Tests
    • Added coverage for JWT expiry decoding behavior.