fix(host-service): force tunnel reconnect on inbound silence without waiting for onclose

Closed
AN
3mo ago
+195-5

Fixes #6229

Bug

A persistent Linux host repeatedly goes offline in the Superset control plane while the VPS, network, host-service process, and terminals all stay running. The host logs no inbound traffic for 77s, forcing reconnect — and then never reconnects until systemctl --user restart superset-host.

Root cause (confirmed by the triage bot and a contributor in the issue thread):

tunnel-client.ts detects relay silence via a watchdog, but its only recovery mechanism was a graceful WebSocket.close(4002, …) — and every reconnect after an established connection was scheduled exclusively from the onclose handler. On a blackholed TCP path (packets dropped rather than reset — the stuck ESTAB with a ~39 KB unacked send queue seen in the report), the close frame can never be flushed, the closing handshake never completes, onclose never fires, and scheduleReconnect() is never called. The client parks in CLOSING forever.

Fix

forceReconnect(socket, code, reason) in tunnel-client.ts:

  • orphans the stale socket immediately — this.socket = null, connecting = false, stopWatchdog(), cleanupChannels() — and calls scheduleReconnect() directly, without waiting for onclose;
  • mirrors the existing connect-timeout path (connect()'s deadline already did close + null + scheduleReconnect);
  • the watchdog now calls it instead of the bare socket.close();
  • the identity guard at the top of onclose (if (this.socket !== socket) return) means the stale socket's late close (if it ever arrives) is ignored — no double-scheduling, no teardown of a newer socket.

Validation

  • bun test on tunnel-client.test.ts: 3/3 pass using a fake WebSocket whose close() never fires onclose (the #6229 failure mode) — forceReconnect schedules the reconnect anyway; stale-socket and post-close no-op cases covered.
  • tsc --noEmit on packages/host-service: 0 errors
  • Biome: clean

Note: tunnel-client-v2.ts may share the pattern; this PR scopes to the v1 client the report and triage identified.


Summary by cubic

Force reconnects the tunnel on inbound silence and ignores events from orphaned sockets. Previously the watchdog waited for onclose after close(4002); on blackholed TCP, onclose never fired and the client stayed in CLOSING.

  • Adds forceReconnect() to orphan the current socket, stop the watchdog, clean channels, and scheduleReconnect() immediately; the watchdog now calls this on inbound silence.
  • Guards onopen, onmessage, onerror, and onclose by socket identity so late events from orphaned sockets don’t touch state or double-schedule.
  • Adds tests that drive the real connect() path with a global FakeWebSocket, stub scheduleReconnect to avoid timer leaks, verify reconnect without onclose, ignore stale sockets and late messages, and close the client to suppress the pending connect deadline.

Summary by CodeRabbit

  • Bug Fixes

    • Improved automatic reconnection when a WebSocket becomes unresponsive.
    • Replaced stale connections promptly, even when the close handshake does not complete.
    • Prevented outdated connections from interfering with active sessions or reconnection attempts.
    • Ensured cleanup is skipped after the client has been closed.
  • Tests

    • Added regression coverage for unresponsive WebSockets, stale callbacks, reconnect scheduling, and late messages from orphaned connections.