Bun.listen / Bun.connect: send the whole end(data) chunk before the socket closes
RO
1mo ago+1484+1484+0123456789012345678901234567890123456789-85-85-01234567890123456789
Stacked on #44093.
Problem
socket.end(data)on aBun.listen/Bun.connectsocket drops what one send does not take.end(Buffer.alloc(32 << 20))returned 2633835 and the peer lost 30920597 bytes, with a clean FIN and noerror.- A queue alone loses the tail when the peer's FIN comes first: the usockets loop closes the socket right after
on_end(packages/bun-usockets/src/loop.c).end(8 MiB)delivered 2634240 bytes.
Fix
end(data)queues the unsent tail.on_writablesends it, and the close follows the last byte.- Bun sockets always open half-open in usockets.
NewSocket::on_endappliesallowHalfOpen: false: close at once when nothing is queued, after the tail when something is. - A fatal write error reaches
error, orclose(socket, err)without anerrorhandler. - Verified:
test/js/bun/net/socket.test.ts, 136 new tests, 107 fail on the base.
Background
- uSockets has no write queue:
us_socket_writeis onesend(2), and the caller keeps the rest. allow_half_openleaves a socket open after the peer's FIN. The owner decides when to close, as uWS HTTP andnode:netdo.- Considered a flag that stops the loop's close only while a tail is queued. It keeps two close paths and misses TLS bytes in the ciphertext spill.
Downsides
- A peer that sends its FIN and never reads holds the socket and the tail until it closes: 100 such peers held 100 fds and 615 MiB. Before, the server closed at once.
- Behaviour change:
end(data)returns the chunk size (8388608, was 2633835). - Size: release binary +4096 B,
Listener+8 B. Syscalls per connection with nothing queued: unchanged.
Notes
Repros (linux-x64)
- Plain case, bun 1.4.3: a
node:netserver counts bytes, the client doesBun.connect({ socket: { open(s) { s.end(Buffer.alloc(32 << 20, 97)) } } }). Before:{"sent":33554432,"end_returned":2633835,"received":2633835,"lost":30920597}. After:lost: 0. - FIN first: a
Bun.listenserver answerss.end(Buffer.alloc(8 << 20))indata()to anode:netclient that didc.end(request)and keeps reading. Release builds, 3 of 3 each, base to head: tcp 2634240 to 8388608, TLS 1.3 2605056 to 8388608, TLS 1.2 2572288 to 8388608. node v26.3.0 andnode:neton bun: 8388608. - A client that destroys the connection after the first chunk (AF_UNIX): the server gets
errorEPIPE with syscallwrite, thenclose. Before: a clean close andend()returned 219264. node reports ECONNRESET there.
Measurements (release builds, base = #44093, head = 6927ed38a3)
- Counter: an
LD_PRELOADshim that counts libc calls (no strace on the machine). Per connection = (count at N=300 - count at N=100) / 200, two runs each. - Default Bun socket, peer closes first, nothing queued, both sockets in one process:
epoll_ctlADD 2, MOD 1, DEL 2,send1,recv3,shutdown1,close2,TCP_CORK0. Same on base and head, with and without adrainhandler. - One short
write()and a later writable event, nothing queued byend():setsockopt(TCP_CORK)per connection 1 to 1 without adrainhandler, 2 to 2 with one. - Release binary 80827976 B to 80832072 B. The binary has no symbols for the
NewSocketfunctions, so there is no size per function. size_of:NewSocket<false>andNewSocket<true>504 B before and after (the newu16errno fits in padding),Listener240 to 248 B. No C struct changes.- Peers that FIN and never read, 100 connections,
end(8 MiB)each: head holds 100 server sockets, 200 fds over the start and +615 MiB RSS until the peers close, then 0 sockets and 0 fds over the start. Base: 0 server sockets, +9 to +48 MiB. - The queue moves its remainder after each partial send (existing
internal_flush, shared withnode:netwrites). Debug build, sends clamped by fault injection: 4 MiB in 16 KiB steps 254 flushes, 502 MiB moved. 4 MiB in 4 KiB steps 1,022 flushes, 2,038 MiB. 8 MiB in 4 KiB steps 2,046 flushes, 8,172 MiB. Real loopback:end(32 MiB)7 flushes, 55 MiB. An offset-based queue is a separate change. - Not measured: allocations and queue appends per connection for
end(small). By the code, the append runs only after a short send.
Behaviour changes in full
end(data)returns the chunk size, not the count one send took.write()andend()return-1afterend(), as the typedoc says.- Without an
errorhandler, a fatal write error arrives asclose(socket, err)with syscallwrite. Before this branch the process could exit 1 there with an uncaught EPIPE.
What the on_end step does
- No
endhandler: with a queue it setsEND_AFTER_FLUSHand keeps the socket, otherwise it closes as before. - With an
endhandler, after the handler returns: nothing when the socket is closed orallowHalfOpenis true, otherwise it ends the socket once the queue is empty. - The user's option is stored as
Flags::ENDS_ON_PEER_FIN, set only byBun.listenandBun.connect. A socket from any other constructor (node:net,node:tls,node:http2native handles) stays half-open as today. Named pipes and duplex upgrades keep today's path. - The raw half of an
upgradeTLSpair keeps one send perend(data)and its old return value: no writable event reaches it. on_writablereturns early as on main when there is nodrainhandler, nothing is queued, and noend()is pending.
Tests
socket.test.tshas 246 tests, 110 on the base. With the base'ssrc/: 108 fail (107 new ones, plusshould not call drain before handshake, which needs the public internet). With this branch: 239 pass, 6 skip, that same 1 fails.- The 29 new tests that pass on the base pin behaviour that was already right:
allowHalfOpen: truerows, and a FIN with nothing queued. - Matrix: tcp, unix, TLS 1.2, TLS 1.3, with and without an
endhandler,end(data)called indata,openandend, listen side and connect side,allowHalfOpenfalse and true. Also: TLS with the last batch in the ciphertext spill (fault injection, debug and ASAN lanes),upgradeTLShalves, an early client disconnect on a server without anerrorhandler, andreload()before the FIN. - Also run on a debug build:
test/js/bun/net/{tcp-server,socket-syscall-fault}.test.ts,test/js/node/net/{node-net-allowHalfOpen.test.js,node-net.test.ts,node-net-server.test.ts},test/js/node/tls/{node-tls-socket-allow-half-open-option,node-tls-raw-end,node-tls-server,node-tls-connect,tls-syscall-fault}.test.ts,test/js/node/http/{node-http-server-socket-end-drain,node-http-backpressure}.test.ts, the ninetest-net-half-open-peer-reset-*.mjs, three node parallel half-open scripts, andtest/integration/bun-types/bun-types.test.ts.node-net.test.tsfails 10 tests that also fail on a build of main on the same machine.
Not in this PR
- A bound on a tail that a peer pins. A peer that sent its FIN and does not read holds the socket and the queued tail with no default limit. The control today is the
timeouthandler: accepted sockets get a 120 s idle timer, and the handler can callterminate(). Without a handler nothing closes the socket.node:netholds a pending write the same way. - A tail that cannot be delivered after a reset or hangup still closes clean when the loop closes the socket. That is the next PR of the stack.
- Windows named pipes still close the writer on EOF.
- A paused socket with an unread request and FIN, then
end(data): the peer can see ECONNRESET. Every design that was weighed has this. #35947 asks the same question.
Review history
- Bot review rounds found, and this branch fixed: a tail queued after a fatal first send, data-less
write()/end()afterend()returning 0,draindispatched after the flush closed the socket (inon_writableand in the flush afteropen()), and the FIN-first truncation that led to the half-open design.
no test proof · iteration 0 · platform-specific test(s) that do not run on this machine, deferring to CI, which covers all platforms: test/js/bun/net/socket.test.ts