Bun.listen / Bun.connect: send the whole end(data) chunk before the socket closes

RO
1mo ago
+1484-85

Stacked on #44093.

Problem

  • socket.end(data) on a Bun.listen / Bun.connect socket drops what one send does not take. end(Buffer.alloc(32 << 20)) returned 2633835 and the peer lost 30920597 bytes, with a clean FIN and no error.
  • A queue alone loses the tail when the peer's FIN comes first: the usockets loop closes the socket right after on_end (packages/bun-usockets/src/loop.c). end(8 MiB) delivered 2634240 bytes.

Fix

  • end(data) queues the unsent tail. on_writable sends it, and the close follows the last byte.
  • Bun sockets always open half-open in usockets. NewSocket::on_end applies allowHalfOpen: false: close at once when nothing is queued, after the tail when something is.
  • A fatal write error reaches error, or close(socket, err) without an error handler.
  • Verified: test/js/bun/net/socket.test.ts, 136 new tests, 107 fail on the base.

Background

  • uSockets has no write queue: us_socket_write is one send(2), and the caller keeps the rest.
  • allow_half_open leaves a socket open after the peer's FIN. The owner decides when to close, as uWS HTTP and node:net do.
  • Considered a flag that stops the loop's close only while a tail is queued. It keeps two close paths and misses TLS bytes in the ciphertext spill.

Downsides

  • A peer that sends its FIN and never reads holds the socket and the tail until it closes: 100 such peers held 100 fds and 615 MiB. Before, the server closed at once.
  • Behaviour change: end(data) returns the chunk size (8388608, was 2633835).
  • Size: release binary +4096 B, Listener +8 B. Syscalls per connection with nothing queued: unchanged.
Notes

Repros (linux-x64)

  • Plain case, bun 1.4.3: a node:net server counts bytes, the client does Bun.connect({ socket: { open(s) { s.end(Buffer.alloc(32 << 20, 97)) } } }). Before: {"sent":33554432,"end_returned":2633835,"received":2633835,"lost":30920597}. After: lost: 0.
  • FIN first: a Bun.listen server answers s.end(Buffer.alloc(8 << 20)) in data() to a node:net client that did c.end(request) and keeps reading. Release builds, 3 of 3 each, base to head: tcp 2634240 to 8388608, TLS 1.3 2605056 to 8388608, TLS 1.2 2572288 to 8388608. node v26.3.0 and node:net on bun: 8388608.
  • A client that destroys the connection after the first chunk (AF_UNIX): the server gets error EPIPE with syscall write, then close. Before: a clean close and end() returned 219264. node reports ECONNRESET there.

Measurements (release builds, base = #44093, head = 6927ed38a3)

  • Counter: an LD_PRELOAD shim that counts libc calls (no strace on the machine). Per connection = (count at N=300 - count at N=100) / 200, two runs each.
  • Default Bun socket, peer closes first, nothing queued, both sockets in one process: epoll_ctl ADD 2, MOD 1, DEL 2, send 1, recv 3, shutdown 1, close 2, TCP_CORK 0. Same on base and head, with and without a drain handler.
  • One short write() and a later writable event, nothing queued by end(): setsockopt(TCP_CORK) per connection 1 to 1 without a drain handler, 2 to 2 with one.
  • Release binary 80827976 B to 80832072 B. The binary has no symbols for the NewSocket functions, so there is no size per function.
  • size_of: NewSocket<false> and NewSocket<true> 504 B before and after (the new u16 errno fits in padding), Listener 240 to 248 B. No C struct changes.
  • Peers that FIN and never read, 100 connections, end(8 MiB) each: head holds 100 server sockets, 200 fds over the start and +615 MiB RSS until the peers close, then 0 sockets and 0 fds over the start. Base: 0 server sockets, +9 to +48 MiB.
  • The queue moves its remainder after each partial send (existing internal_flush, shared with node:net writes). Debug build, sends clamped by fault injection: 4 MiB in 16 KiB steps 254 flushes, 502 MiB moved. 4 MiB in 4 KiB steps 1,022 flushes, 2,038 MiB. 8 MiB in 4 KiB steps 2,046 flushes, 8,172 MiB. Real loopback: end(32 MiB) 7 flushes, 55 MiB. An offset-based queue is a separate change.
  • Not measured: allocations and queue appends per connection for end(small). By the code, the append runs only after a short send.

Behaviour changes in full

  • end(data) returns the chunk size, not the count one send took.
  • write() and end() return -1 after end(), as the typedoc says.
  • Without an error handler, a fatal write error arrives as close(socket, err) with syscall write. Before this branch the process could exit 1 there with an uncaught EPIPE.

What the on_end step does

  • No end handler: with a queue it sets END_AFTER_FLUSH and keeps the socket, otherwise it closes as before.
  • With an end handler, after the handler returns: nothing when the socket is closed or allowHalfOpen is true, otherwise it ends the socket once the queue is empty.
  • The user's option is stored as Flags::ENDS_ON_PEER_FIN, set only by Bun.listen and Bun.connect. A socket from any other constructor (node:net, node:tls, node:http2 native handles) stays half-open as today. Named pipes and duplex upgrades keep today's path.
  • The raw half of an upgradeTLS pair keeps one send per end(data) and its old return value: no writable event reaches it.
  • on_writable returns early as on main when there is no drain handler, nothing is queued, and no end() is pending.

Tests

  • socket.test.ts has 246 tests, 110 on the base. With the base's src/: 108 fail (107 new ones, plus should not call drain before handshake, which needs the public internet). With this branch: 239 pass, 6 skip, that same 1 fails.
  • The 29 new tests that pass on the base pin behaviour that was already right: allowHalfOpen: true rows, and a FIN with nothing queued.
  • Matrix: tcp, unix, TLS 1.2, TLS 1.3, with and without an end handler, end(data) called in data, open and end, listen side and connect side, allowHalfOpen false and true. Also: TLS with the last batch in the ciphertext spill (fault injection, debug and ASAN lanes), upgradeTLS halves, an early client disconnect on a server without an error handler, and reload() before the FIN.
  • Also run on a debug build: test/js/bun/net/{tcp-server,socket-syscall-fault}.test.ts, test/js/node/net/{node-net-allowHalfOpen.test.js,node-net.test.ts,node-net-server.test.ts}, test/js/node/tls/{node-tls-socket-allow-half-open-option,node-tls-raw-end,node-tls-server,node-tls-connect,tls-syscall-fault}.test.ts, test/js/node/http/{node-http-server-socket-end-drain,node-http-backpressure}.test.ts, the nine test-net-half-open-peer-reset-*.mjs, three node parallel half-open scripts, and test/integration/bun-types/bun-types.test.ts. node-net.test.ts fails 10 tests that also fail on a build of main on the same machine.

Not in this PR

  • A bound on a tail that a peer pins. A peer that sent its FIN and does not read holds the socket and the queued tail with no default limit. The control today is the timeout handler: accepted sockets get a 120 s idle timer, and the handler can call terminate(). Without a handler nothing closes the socket. node:net holds a pending write the same way.
  • A tail that cannot be delivered after a reset or hangup still closes clean when the loop closes the socket. That is the next PR of the stack.
  • Windows named pipes still close the writer on EOF.
  • A paused socket with an unread request and FIN, then end(data): the peer can see ECONNRESET. Every design that was weighed has this. #35947 asks the same question.

Review history

  • Bot review rounds found, and this branch fixed: a tail queued after a fatal first send, data-less write()/end() after end() returning 0, drain dispatched after the flush closed the socket (in on_writable and in the flush after open()), and the FIN-first truncation that led to the half-open design.

no test proof · iteration 0 · platform-specific test(s) that do not run on this machine, deferring to CI, which covers all platforms: test/js/bun/net/socket.test.ts