fix(relay): un-wedgeable tunnel supervision + SDK relay resolution

Merged
SA
3mo ago
+682-46

The bug class

Three field incidents this week (two user reports + one detailed feedback doc) shared one signature: host process alive and API-reachable, relay tunnel connected to no relay, zero reconnect attempts anywhere, fixed only by manually restarting the host service. One reporter's watchdog log showed a 4.4-hour rescue on a 75-second timeout.

Reproduced mechanism (v2, the current fleet)

Partysocket awaits the async urlProvider before creating a socket, so its connectionTimeout cannot cover it. A getAuthToken() that hangs (network call to an unreachable API, no timeout) stalls the provider forever — the retry cycle dies with no timer running anywhere. Reproduced against the live relay: old client makes 1 attempt then goes silent permanently; fixed client keeps cycling indefinitely. (A throwing provider was already survivable — the hang is the killer.)

The v1 client has the sibling defect: its watchdog early-returns unless the socket is OPEN, and its silence recovery calls close() and waits for onclose — which a dead peer never delivers (undici has no close-handshake timeout), leaving CLOSING sockets supervised by nothing.

The fix — one invariant

At every moment, some timer is armed that leads back to a healthy socket.

  • v2: 15s timeout on each url-provider step (hang → fallback, cycle moves on); provider never rejects; watchdog kicks control.reconnect() when the control socket sits outside OPEN past 60s.
  • v1: watchdog supervises every state (OPEN silence; sockets stuck non-OPEN past 30s; a dead reconnect chain) and abandons sockets by detaching immediately — the onclose guard makes late events no-ops.
  • Both: relay URL re-resolved via the API on every reconnect attempt (fallback: last known). A server-side relay move now converges on the next reconnect, not the next process restart — previously the URL was pinned once per process lifetime, which is why the relay migration only moved hosts when machines rebooted.

Verification

  • Live-relay harness, hanging provider: old 1 attempt/45s (wedge), new 3 attempts/45s and ongoing
  • Live-relay harness, throwing providers (both): 10 attempts/30s, cycle survives
  • @superset/host-service typecheck + test suite green

Ships on the desktop release cadence — candidate for the next patch release.

Summary by cubic

Prevents the tunnel from wedging and ensures host-routed SDK calls use the active relay. Previously v2 could stall on a hanging URL provider and v1 could sit in CONNECTING/CLOSING waiting for a peer; now both always progress back to a healthy socket, and we emit limited Sentry signals that the supervision worked.

  • v2: 15s timeout on each url-provider step with fallback; provider never rejects; watchdog kicks partysocket if control isn’t OPEN for >60s.
  • v1: Watchdog supervises every state; abandons stuck sockets immediately (no close-handshake wait); restarts dead reconnect chains.
  • Hosts + SDK: Re-resolve the relay URL via the API (hosts: every reconnect; SDK: before host-routed calls with 60s cache and single-flight), falling back to the last known/configured URL; explicit relayURL or SUPERSET_RELAY_URL still wins.
  • Observability: Report one Sentry “tunnel rescue” per reason per hour (v1_stuck_socket, v1_dead_reconnect_chain, v2_token_fetch_failed, v2_control_stuck) to verify field impact.
  • Rollout: No config changes. Expect slightly more retry traffic under sustained outages.

Summary by CodeRabbit

  • Bug Fixes
    • Improved tunnel reliability by refreshing relay connections when needed.
    • Added safeguards for delayed or failed relay URL and authentication lookups.
    • Reuses the last known relay when resolution fails.
    • Automatically recovers connections that become unresponsive or remain stuck during setup.
    • Preserves reconnection behavior when sockets close or stop receiving data.