ServerWebSocket.terminate(): close a TLS socket without waiting for the peer

Merged
RO
1mo ago
+164-1

Problem

  • With Bun.serve({ tls }), ws.terminate() waits for the peer's TLS close_notify. A peer that does not read never sends it: the close handler does not run, the fd stays open, and message() keeps firing. Each inbound frame re-arms the idle timeout, so nothing bounds the wait. Plain ws:// closes at once.
  • The cause is WebSocket::close() (packages/bun-uws/src/WebSocket.h:76): us_socket_close(s, 0, nullptr). On TLS, code 0 is the graceful close (us_internal_ssl_close, packages/bun-usockets/src/crypto/openssl.c:2194). forceClose() with no reason (a reserved opcode) passes the same code.

Fix

  • WebSocket::close() closes with FAST_SHUTDOWN (a FIN, no close_notify). If the socket is still open, usockets parked the close behind its ciphertext spill, and close() closes again with CONNECTION_RESET (the idiom of #39548). forceClose() with no reason calls close().
  • Correct because every caller relies on the close event having run on return (CloseCode, src/uws_sys/us_socket_t.rs:23). Plain sockets treat codes 0 and 2 alike. A peer that keeps up sees no change. Under send backpressure the close is now a reset (see Notes).
  • Not changed: the other code 0 closes in uWS (HTTP, HTTP/2), listed in the Notes.
  • Verified: new test/js/bun/websocket/websocket-server-forced-close.test.ts, the 3 wss cases fail without the fix. Also test/js/bun/websocket/, test/js/web/websocket/, test/js/first_party/ws/.

Background

  • us_socket_close(s, code, reason) closes the fd and runs the close handler. On TLS, code 0 sends close_notify and waits for the peer, code 1 resets, and code 2 sends a FIN but waits while a spill is pending.
  • A spill is TLS ciphertext that bun reported as written, but the kernel did not accept yet.
  • uWS close() is the forced close (1006, no Close frame). end(), the graceful close, does not change.
  • #38243 fixed this wait for the client WebSocket.
Notes

Repro (1.4.3, canary c6b7fcb5b, and a debug build of main b8eacead4d). A wss:// server sends one message. The client calls pause() and sends one message back. The server calls ws.terminate() in message().

wss, before:  0.0s SERVER terminate(), readyState 3
              12.0s done waiting                       (close handler never ran)
ws,  before:  0.0s SERVER close handler: 1006 ""
              0.0s SERVER terminate(), readyState 3
wss, after:   0.1s SERVER close handler: 1006 ""
              0.1s SERVER terminate(), readyState 3
  • With idleTimeout: 8 and a silent peer, the unfixed server closes at 8.0 s from the idle timer, with the reason "WebSocket timed out from inactivity".
  • With idleTimeout: 4 and a paused peer that sends one frame each second, the unfixed server never closes. In 14 s it delivered 14 messages to message() after terminate(), all with ws.readyState === 3. The uWS parser stops only when us_socket_is_closed() or isShuttingDown is true, and a deferred TLS close sets neither.
  • Protocol error: a paused Bun.connect({ tls }) client sends a masked frame with opcode 3, and 300 ms later a valid text frame. Before: no close, and the text frame reaches message(). After: close 1006 at once. The forceClose() calls that pass a reason already close at once, because the reason length is the close code and it is never 0.

A peer that keeps up sees no change. Matrix: ws.send(payload); ws.terminate() from a handler and from a timer, payload 100 B, 64 KB, 300 KB and 4 MB, 10 runs each, a Bun WebSocket client that reads. All 16 cells are the same for ws and wss, before and after: the same messages arrive, then close 1006, and no error event. A node ws client also reports close 1006 with no error event, before and after.

A peer under send backpressure gets a reset. A spill does not prove a dead peer. It means the last flush hit a full kernel buffer, which also happens with a slow peer that still reads. If the first close cannot drain the spill, the second close resets: the kernel drops its send buffer and the peer sees ECONNRESET. Before this change that peer got the kernel buffer and the spill, then close_notify, whenever the spill drained. Plain ws:// gives it the kernel buffer and a FIN. The uWS backpressure buffer is lost in every variant, so the stream ends in the middle of a message in every variant. The reset is the documented cost of a close that must finish now (CloseCode::failure names terminate()). The client WebSocket.terminate() (#38243) and Socket.terminate() reset on every close. A FIN that keeps the kernel buffer needs a usockets close that drops the spill without a wait, which is the durable fix that #39548 names.

Why a FIN first and not a reset for every TLS close. With a node TLS client, a server reset gives error: ECONNRESET. A bare FIN gives end, then close with no error, which is what ws:// gives today. npm ws 8.18.3 does the same: its terminate() is socket.destroy(), and the comment in us_internal_ssl_close records that node's destroy sends a bare FIN.

Why the second close. A build with FAST_SHUTDOWN only still hangs in the backpressure test (paused client, the server sends 64 KB messages until send() returns -1, then terminate()). us_internal_ssl_close defers codes 0 and 2 until the spill drains. The check after the first close detects the deferral and does not predict it. usockets first tries to drain the spill, so a peer that recovered still gets a FIN. A check of us_socket_ssl_spill_pending() before the close is not equivalent: it resets a peer that has recovered. #39548 records that.

The read of the socket after the first close is safe. us_internal_socket_close_raw links a socket into the loop's closed list only after the close handler returns, so JS that re-enters the event loop from the handler cannot free it.

Other callers of WebSocket::close() that get the same fix: the open() exception path in ServerWebSocket.rs, the HMR socket (hmr_socket.rs), and DevServer teardown, which has debug_assert!(self.active_websocket_connections.is_empty()) right after it closes every socket.

Not fixed here. These sites have the same exposure. Each needs its own repro and tests, and the durable fix is a usockets close that cannot defer (see the last Background bullet of #39548).

  • AsyncSocket<SSL>::close() (AsyncSocket.h:101) and the HTTP paths that use it. One case is reproduced: Bun.serve({ tls, idleTimeout: 4 }) with a peer that completes the handshake and stops reading. The plain server socket is in FIN-WAIT-2 at 12 s. The TLS server socket is still ESTAB at 40 s.
  • The code 0 closes in Http2Context.h.
  • JSNodeHTTPServerSocket::close() (JSNodeHTTPServerSocket.cpp:90). It already uses FAST_SHUTDOWN, with no second close for a spill.
  • A peer FIN on a TLS WebSocket that still has a spill, and closeOnBackpressureLimit. These are not forced closes.

Order with #35874. That PR replaces the forceClose() calls that pass no reason with a graceful end(1002). If it merges first, the reason.empty() branch here has no caller and can go, and the reserved opcode test must expect the code that PR reports. If this PR merges first, #35874 needs no change to close().

Test file. The test is a sibling file and not part of websocket-server.test.ts. That file has load-related timeouts on a debug build (8 here, in send(), sendText(), sendBinary() and the benchmark). The same tests pass alone. The two terminate() tests fail at once on an unfixed build, with close: 1006 "" missing between message and terminate() returned. The reserved opcode test times out there.

Suites run on a debug+ASAN build: all of test/js/bun/websocket/, websocket.test.js, websocket-pause.test.ts, websocket-client.test.ts, websocket-close-code.test.ts, websocket-close-fragmented.test.ts, websocket-proxy.test.ts, websocket-subprotocol-strict.test.ts, websocket-client-short-read.test.ts, websocket-close-connecting.test.ts, websocket-close-async-dispatch.test.ts, websocket-buffered-amount.test.ts, websocket-server-send-from-drain.test.ts, ws.test.ts, ws-upgrade-events.test.ts, and the pipelined upgrade tests in serve.test.ts. Failures that do not come from this change: two websocket.test.js tests that need ws.postman-echo.com, and tests that spawn a debug build per case and go over 5 s when this machine is loaded ("should connect many times over https", ws-upgrade-events.test.ts). They pass alone.


[human-review] gate passed · iteration 0 · 3 files touched

fails on main (without fix)
ASAN without fix: 3 FAILED
$ BUN_DEBUG_QUIET_LOGS=1 bun scripts/build.ts --profile=debug --quiet test "--reporter=junit" "--reporter-outfile=/tmp/pr_gate.xml" test/js/bun/websocket/websocket-server-forced-close.test.ts
bun test v1.4.3 (c6b7fcb5b)

test/js/bun/websocket/websocket-server-forced-close.test.ts:
(pass) ws: forced close with a peer that does not read > terminate() runs the close handler before it returns [189.90ms]
(pass) ws: forced close with a peer that does not read > terminate() runs the close handler before it returns under backpressure [156.98ms]
(pass) ws: forced close with a peer that does not read > a frame with a reserved opcode closes the socket [139.63ms]
53 |       client.onmessage = () => {
54 |         client.pause();
55 |         client.send("terminate");
56 |       };
57 |       await terminated.promise;
58 |       expect(events).toEqual(["message: terminate", 'close: 1006 ""', "terminate() returned"]);
                          ^
error: expect(received).toEqual(expected)

  [
    "message: terminate",
-   "close: 1006 """,
    "terminate() returned",
  ]

- Expected  - 1
+ Received  + 0

      at <anonymous> (/workspace/bun/test/js/bun/websocket/websocket-server-f
... (truncated)

release without fix: all passed
bun test v1.4.3-canary.1 (fed761ce5)

test/js/bun/websocket/websocket-server-forced-close.test.ts:
(pass) ws: forced close with a peer that does not read > terminate() runs the close handler before it returns [18.47ms]
(pass) ws: forced close with a peer that does not read > terminate() runs the close handler before it returns under backpressure [16.71ms]
(pass) ws: forced close with a peer that does not read > a frame with a reserved opcode closes the socket [16.37ms]
(pass) wss: forced close with a peer that does not read > terminate() runs the close handler before it returns [17.73ms]
(pass) wss: forced close with a peer that does not read > terminate() runs the close handler before it returns under backpressure [11.70ms]
(pass) wss: forced close with a peer that does not read > a frame with a reserved opcode closes the socket [10.75ms]

 6 pass
 0 fail
 6 expect() calls
Ran 6 tests across 1 file. [85.00ms]
__F:0:S:0
passes on PR (with fix)
ASAN with fix: all passed
$ BUN_DEBUG_QUIET_LOGS=1 bun scripts/build.ts --profile=debug --quiet test "--reporter=junit" "--reporter-outfile=/tmp/pr_gate.xml" test/js/bun/websocket/websocket-server-forced-close.test.ts
bun test v1.4.3 (c6b7fcb5b)

test/js/bun/websocket/websocket-server-forced-close.test.ts:
(pass) ws: forced close with a peer that does not read > terminate() runs the close handler before it returns [278.56ms]
(pass) ws: forced close with a peer that does not read > terminate() runs the close handler before it returns under backpressure [237.39ms]
(pass) ws: forced close with a peer that does not read > a frame with a reserved opcode closes the socket [210.23ms]
(pass) wss: forced close with a peer that does not read > terminate() runs the close handler before it returns [211.46ms]
(pass) wss: forced close with a peer that does not read > terminate() runs the close handler before it returns under backpressure [155.50ms]
(pass) wss: forced close with a peer that does not read > a frame with a reserved opcode closes the socket [71.47ms]

 6 pass
 0 fail
 6 expect() calls
Ran 6 tests across 1 file. [2.82s]
__F:0:S:0

release with fix: all passed
$ bun scripts/build.ts --profile=release
[configured] bun-profile → bun (stripped) in 841ms (unchanged)
ninja: Entering directory `/workspace/bun/build/release'
[1/128] gen generated_host_exports.rs
generated_host_exports.rs: 121 exports (host=5, lazy=10, generic=106, rust=0); 242 extern-C blocks audited
[2/128] gen JSSink.{cpp,h,lut.h,rs}
generated_jssink.rs: 7 sinks, 84 exported symbols
Generating /workspace/bun/build/release/codegen/JSSink.lut.h from /workspace/bun/build/release/codegen/JSSink.lut.txt
[3/128] gen cpp.rs (cppbind)
[4/128] gen JS modules (bundle-modules)
Preprocess modules (11568ms)
Bundle modules (91ms)
Postprocesss modules (179ms)
Bundle Functions (587ms)
Generate Code (41ms)

[12.48s] Bundled "src/js" for production
  2600 kb
  197 internal modules
  13 native modules
  50 internal functions across 16 files
[4/26] cargo bun_runtime → libbun_runtime.a
warning: binary `bun_shim_impl` should have a kebab-case name
   |
 1 | /workspace/bun/build/release/rust-target/.../bun_shim_impl
   |                                              ^^^^^^^^^^^^^
   |[
... (truncated)
diff hotspot
packages/bun-uws/src/WebSocket.h                   |   8 +-
 packages/bun-uws/src/WebSocketContext.h            |   5 +
 .../websocket-server-forced-close.test.ts          | 152 +++++++++++++++++++++
 3 files changed, 164 insertions(+), 1 deletion(-)

gate history · 2 passed · 0 rejected · iteration 0

evidence per changed file
file                                                      reads  edits  tests
packages/bun-uws/src/WebSocket.h                              4      5     21
packages/bun-uws/src/WebSocketContext.h                       2      3     20
…/js/bun/websocket/websocket-server-forced-close.test.ts      1      2     20