fix(relay): never route dials to a replaced host socket after hibernation

Merged
HA
3h ago
+57-5

Summary

  • The relay could keep sending a host's requests to a control socket the host had already abandoned. Every relayed tRPC call to that host then failed with Request timed out after 65 s, for minutes at a time, until the liveness alarm closed the dead socket.
  • hostConn() now skips host sockets the relay has retired (replaced or stale). The retired mark is stored in the socket's connection state, which survives hibernation. readyState doesn't.

Why / Context

A customer (host e89c80f0…, host-service 2.0.5) reported workspaces.create, list and terminal calls stalling for about 66 s in windows on 2026-11-10 (17:55:53–17:59:03 and 18:10:00–18:11:19 UTC), while their control tunnel stayed connected. Relay logs for that host's Durable Object show:

  • In both windows no dial-back arrived from the host, yet the live control socket's pings reached the object every 30 s.
  • The relay still logged host stale, closing (17:57:07, 17:59:23, 18:10:23). A ping only refreshes liveness when it comes from hostConn(), so hostConn() was returning a different socket.
  • Dial-backs resumed 2 s after the stale close at 17:59:23, once the dead socket was closed.

The same 65 s 502 shows up across the fleet: 6,265 of 1.87M relayed tRPC calls between 15:00 and 19:00 UTC that day (about 0.34%, roughly 1,500 an hour), across many hosts and colos. This PR should cover a large share of that; it has not been confirmed per host.

How It Works

Root cause, reproduced on the real Workers runtime:

  1. A host's control socket dies without a close frame (WebSocket 1006 on the host side), and the host reconnects.
  2. onConnect closes the old socket. The peer is gone and never answers, so the socket stays CLOSING.
  3. The Durable Object hibernates and wakes. The runtime now reports the old socket as OPEN again.
  4. hostConn() returned the first OPEN host socket, so every stream:dial went to the dead socket. The exchange hit the 65 s timer, and isConnected(), also fooled, turned that into BAD_GATEWAY "Request timed out".

Fix: retireHost() writes retired: true into the host socket's state before closing it. That state lives in the WebSocket attachment and survives hibernation. hostConn() returns the first host socket that isn't retired. Both the replace path in onConnect and the stale path in onAlarm retire sockets.

Manual QA

Ran the real HostTunnel class as a throwaway Worker (relay-repro, its own Durable Object namespace; deleted afterwards) on Cloudflare. Socket A connected through a local TCP proxy that was then frozen to imitate a dead peer (no FIN/RST), socket B connected directly, and one relayed call ran every 25 s while B pinged every 30 s.

  • Before (main): after a wake, A read OPEN and hostConn() returned A. The call failed with reason: timeout after 65,008 ms; it recovered only after the stale alarm closed A.
  • After (this branch): A read OPEN after wakes at t+50/75/100/125/150 s, but hostConn() stayed on B, and every call succeeded in 0.2–2.4 s.
  • Local wrangler dev can't show this; it never hibernates the object. A replaced socket correctly reads CLOSING there.

Testing

  • bun test in apps/relay: 20 pass. New test: a replaced control socket that reads open again never receives dials. It fails on main (the dial goes to replaced) and passes here.
  • bun run typecheck in apps/relay
  • bunx biome check on the changed files

Known Limitations

  • A retired socket that comes back as OPEN is now ignored but not closed again. It stays attached until the runtime notices the peer is gone; production logs show that happens within minutes. It only costs a slot, and it gets retired again on the next host connect.

Follow-ups

  • Fail a relayed HTTP exchange as soon as its dial socket closes, instead of waiting for the 65 s timer (HostTunnel.handleGone ignores dial sockets that belong to an HTTP exchange).
  • Fail fast, in about 10 s, when no dial-back arrives at all.
  • Report to Cloudflare that a hibernatable WebSocket closed by the server but never acknowledged reads OPEN after the object wakes. Their docs only say it may stay CLOSING.

Risks / Rollout

  • Risk: low. It only changes which host socket is chosen; sockets the relay never retired behave as before.
  • Rollout: merging deploys the relay, which restarts every HostTunnel Durable Object and drops every host's control socket once. Hosts reconnect within seconds. Prefer a quiet hour.
  • Rollback: revert this commit; the same deploy path applies.

Summary by cubic

Fixes the relay routing a host's relayed tRPC calls to a control socket the host had already replaced, which made them time out after 65 seconds until the stale alarm closed the dead socket. hostConn() now skips sockets marked retired in their connection state, which survives hibernation; readyState can't be trusted because Cloudflare reports a server-closed socket as OPEN again after the Durable Object wakes.

  • retireHost() marks a host socket retired in its state before closing it, on both the replace and stale paths.
  • Adds a regression test that fails on main and passes here.
  • Verified on the real Workers runtime: after a wake the replaced socket reads OPEN, but dials only reach the live socket.

Summary by CodeRabbit

  • Bug Fixes
    • When a host reconnects and replaces an existing connection, the previous connection is closed and subsequent proxy traffic is routed through the new connection.
    • Retired and stale host connections are excluded from proxy requests, helping ensure traffic is sent only through an active host connection.