child_process: keep the parent alive for un-read stdio pipes after unref()
Problem
- Since bun 1.4.0,
node:child_processspawn(cmd).unref()with piped stdio lets the parent exit at once.SIGPIPEthen ends the child at its next stdout write. Node and bun 1.3.14 wait for the pipes. - #34971 made these pipes
lazy. Nothing polls a pipe until JS reads it, so afterunref()it holds nothing.
Fix
unref()queuesread(0)for stdout and stderr withsetImmediate. That read holds the loop until the pipe closes or the stream is full.spawn()creates every stdio stream. Streamref()andunref()set one flag, as in Node (from #44590).- Verified:
test/js/node/child_process/child_process.test.ts(11 of 18 new rows fail on 1.4.3), 102 of 106 upstream files. - Self-reviewed: 38 concerns raised, 37 addressed. Rejected: running the rows under Node in CI (needs a dual-runtime fixture).
Background
Bun.spawn({ lazy: true })reads a piped stdout or stderr only when JS reads.node:child_processalways passes it.read(0)starts aReadable's read and consumes nothing. Node'snet.Socketcalls it at construction.NativeReadable, the class of these streams, countedref()calls.- Considered the read at creation: it changes every spawn. #36316's review asks for fd-backed
net.Socketstdio, so this is the stopgap.
Downsides
- After
unref()the parent waits for the pipes, as in Node. It waits forever if a child floods an un-read stdout with stderr open. ref(); ref(); unref()on a child's stdout or stderr no longer holds the loop.unref()on two un-read pipes costs 2 immediates, 4 syscalls and 124 KB. A flooded stream holds 180 KB (Node: 64 KB).
Notes
What this revision changed. The earlier head started the read when node:child_process created the stream, and changed FileReader::on_start so that the start only armed the poll unless the writer had hung up. That put exit before data for a launcher that exits and leaves a background process on the pipe: spawn('sh', ['-c', 'echo hi; sleep 0.2 &']) gave "exit, data" 20 of 20 times on that head, and "data, exit" 20 of 20 on Node v26.3.0, on bun 1.4.3 and on this head. That state is the normal case: on bun 1.4.3 (release) the launcher has already exited when spawn() returns in 138 of 200 spawns. Now:
src/runtime/webcore/FileReader.rsandsrc/io/PipeReader.rsare byte-identical to main.Bun.spawn({ lazy: true })is untouched.- Nothing reads a pipe at spawn. The read starts from
unref(), the only place where an un-read pipe needs its own hold: a child that is not unref'd holds the loop through its process handle, and#handleOnExitresumes the streams when it exits. unref()queues the read withsetImmediateand does not run it on its own stack. A read on the spot takes what the child wrote so far, and the rest arrives as a seconddataevent: 18 of 20 runs forsh -c 'echo one >&2; sleep 0.03; echo two >&2'with aspawnSyncafterunref(). With the queued read: 0 of 30.- The queue is
setImmediateand notprocess.nextTick. bun does not run a tick that anunhandledRejectionhandler queues when nothing else keeps the loop alive (Node runs it). Aspawn().unref()in such a handler left after 20 to 31 ms with the tick and waits 728 to 795 ms with the immediate (Node 707 ms). The function is taken once at module scope, so mock timers that replaceglobalThis.setImmediatedo not hold the read back. That holds for mocks enabled afternode:child_processfirst loads.node:testmock timers enabled before that load take the read, and the parent exits as on 1.4.x (Node waits). WithglobalThis.setImmediatedeleted before that load,unref()throws aTypeError.src/js/node/test.ts(realSetTimeout) andsrc/js/node/timers.promises.tscapture their timers the same way. A private reference to the native function would close the window for one more generated binding entry. A review asked for it, and this PR does not add it. test-child-process-prototype-tampering.mjsis the upstream text. The backpressure test keeps its 1 MiB.- The Bun.spawn-level row for the removed start rule is deleted.
ref()andunref()on aNativeReadableforward to the native reader and return the stream. The JS counter is deleted. A review found thatchild.unref(); stdout.ref(); stdout.unref()held the parent on the earlier head, where Node and bun 1.4.3 exit:spawn()leaves the counter at 1, so the pair never reached the native flag, andunref()now reads the pipe. The change is thenative-readable.tshunk of #44590, with credit to its author in the commit. The two PRs can land in either order. The same class backsReadable.fromWeb()of a native stream, so the count is gone there too.
Repro. Parent: spawn('sh', ['-c', 'sleep 0.4; echo progress; sleep 0.4; echo done > MARKER']).unref(), default stdio, nothing else to do. node v26.3.0: the parent leaves after about 840 ms and MARKER exists. bun 1.4.3: about 20 ms, no MARKER. This branch: MARKER exists, 3 of 3 runs. With stdio: "ignore" every runtime lets the parent leave and the child finishes. Found by a regression sweep. No user report is known.
What holds the parent. Linux x64. Each row spawns sh -c 'sleep 0.3; echo progress; sleep 0.3; echo done > MARKER' after a warm-up spawn. The cell is the time from spawn() to the parent's exit, then whether the child reached its last step. This branch is a debug build, so its short times are longer than a release build's.
| parent does | node v26.3.0 | bun 1.4.3 | this branch |
|---|---|---|---|
stdio: "ignore", c.unref() | 4 ms, yes | 4 ms, yes | 17 ms, yes |
| nothing | 606 ms, yes | 607 ms, yes | 664 ms, yes |
c.unref() | 606 ms, yes | 3 ms, no | 680 ms, yes |
c.unref(), stdio [ignore, pipe, ignore] | 620 ms, yes | 1 ms, no | 629 ms, yes |
c.unref(), stdio [ignore, ignore, pipe] | 610 ms, yes | 3 ms, yes | 622 ms, yes |
c.unref(), stdio [pipe, ignore, ignore] | 2 ms, yes | 7 ms, yes | 20 ms, yes |
c.unref(), stdio [pipe] | 610 ms, yes | 2 ms, no | 642 ms, yes |
c.unref(), child writes nothing | 604 ms, yes | 1 ms, yes | 665 ms, yes |
c.unref(), stdout.unref(), stderr.unref() | 6 ms, no | 4 ms, no | 38 ms, no |
stdout.unref(), stderr.unref(), then c.unref() | 6 ms, no | 13 ms, no | 46 ms, no |
stdout.unref(), stderr.unref(), no c.unref() | 605 ms, yes | 607 ms, yes | 654 ms, yes |
c.unref(), stdout.unref() only | 608 ms, yes | 2 ms, no | 616 ms, yes |
all three unref, then stdout.ref() | 619 ms, yes | 1 ms, no | 645 ms, yes |
all three unref, then c.ref() | 628 ms, yes | 627 ms, yes | 650 ms, yes |
c.unref(), stdout.pause() | 622 ms, yes | 2 ms, no | 689 ms, yes |
c.unref(), stdout.resume() | 621 ms, yes | 607 ms, yes | 645 ms, yes |
c.unref(), stdout.on("data") | 622 ms, yes | 606 ms, yes | 644 ms, yes |
c.unref(), destroy stdout and stderr | 13 ms, no | 2 ms, no | 32 ms, no |
c.unref(), both unref on the next tick | 6 ms, no | 6 ms, no | 28 ms, no |
c.unref(), child floods stdout (4 MiB) | not within 12 s | 2 ms | not within 12 s |
same, stdio [ignore, pipe, ignore] | 11 ms | 9 ms | 68 ms |
| same, all three unref | 3 ms | 2 ms | 34 ms |
Bold cells are where 1.4.3 differs from Node. "no" means SIGPIPE ended the child at echo progress.
ref() and unref() on the streams. The calls below run on both stdout and stderr of a child that sleeps 2.5 s. Node's handles keep one ref flag. NativeReadable counted calls until the last commit of this PR.
| calls on each stream | node v26.3.0 | bun 1.4.3 | this branch, counter | this branch |
|---|---|---|---|---|
| none | holds | exits | holds | holds |
ref() | holds | exits | holds | holds |
unref() | exits | exits | exits | exits |
unref(); ref() | holds | exits | holds | holds |
unref(); ref(); unref() | exits | exits | exits | exits |
ref(); unref() | exits | exits | holds | exits |
ref(); ref(); unref() | exits | exits | holds | exits |
unref(); unref(); ref() | holds | exits | exits | holds |
Each row gives the same result in four positions of the stream calls: before child.unref(), after it, on the next tick, and in setImmediate. That is 32 cells. With the counter 20 equal Node, now 32 do.
Designs weighed.
- Read at creation, inside
spawn()(the first shape of this PR): bun's start reads the pipe on the spot, Node's only arms the handle. A child whose first write is already in the pipe gets it delivered alone, and its next write is a seconddataevent. The unmodified upstreamtest-child-process-prototype-tampering.mjsfails with a two-writepwd, which is why an earlier revision had relaxed it. - Read at creation from a queued callback: it keeps the upstream test, but every piped spawn then has an armed one-shot poll from spawn on. A nested loop tick can drop such an event for good (#33261, #33289, #46545). On 1.4.x a pipe that nothing reads has no poll.
- Arm the poll at start and do not read (
FileReader::on_start):exitcomes beforedatawhen the child has exited before the stream is created. Main's read on the spot is what delivers the data first. libuv gets that order by running child watchers last in a loop iteration. That is a change to how bun dispatches process exits, for everyBun.spawnuser. The follow-up, in order: report a child's exit after the other I/O of the same iteration (#43888 tried one shape and was closed), then a start that only arms, then a cap on what one poll-driven read delivers. - Read in
unref()(this PR): only an unref'd child changes.
Not changed by this PR.
- A pipe of a child that is never unref'd is still not read at spawn. Node arms it there. Its first read still happens on the reader's stack, so
child.stdout.read()in the spawn tick can return bytes where Node returns null. - After
child.unref()Node is also held by thetimeoutkill timer and by the IPC channel. bun drops both. #46906 and #45662 / #36405 cover those. Bun.spawn(...).unref()(bun's own API) still lets the parent leave at once.- An un-read stream still gets no
endorclosebefore the exit path runs. The hold ends at pipe EOF, the events do not move.
Measured. Linux x64, debug build. strace is not available on the machine used here, so syscalls are counted with gdb (catch syscall), attached after the children run. "main" is the same binary with main's node:child_process: no native file differs. The numbers below are from the head before the ref flag change, except where a line says otherwise.
- spawn without
unref(): 0 syscalls added (2recvfrom, 4epoll_ctl, 0pollper spawn on both), 11nextTickentries per spawn against 12, 65.7 JS cells per live child against 68.7 (gdb,heapStats(), N=200 and N=50). - per
unref()with two un-read pipes: +2 immediates, +4 syscalls (2recvfrom, 2epoll_ctl), +4Promise, +123.8 KB held until the pipes close (N=200). On main anunref()adds none of these. node:child_processbundle: 45378 bytes against 45563 on main (debug build).native-readablebundle: 8290 bytes against 8511. No release build was made.- ref flag change, per stream: one property less. Per
ref()orunref()call: one call into the native reader, where the counter made that call only at 0 and 1. Per spawn: the same one call per stream as before. - two
dataevents afterunref(): 0 of 30 (Node 0 of 30, 1.4.3 0 of 30, a synchronousread(0)inunref()18 of 20). databeforeexit, forsh -c 'echo hi'andsh -c 'echo hi; sleep 0.3 &', with and withoutunref(): 160 of 160 (Node 160 of 160, 1.4.3 160 of 160). Not run on a release build of this branch.- upstream, unmodified:
test-child-process-prototype-tampering.mjswith a stand-in for BusyBoxpwd(error line, a gap of 0, 1, 10 or 30 ms, usage text) 120 of 120,test-child-process-can-write-to-stdout.js30 of 30, upstream files 119 of 122 (main: the same 3 fail). With the ref flag change: 102 of the 106test-child-process-*files inparallelandsequential. - matrix: 22 of 22 rows equal to Node, and 32 of 32 stream cells with the ref flag change. Un-read stdout after
unref(): 180224 bytes in the stream (Node 65536). In some runs the read loop holds one more chunk of that size natively. Both come from main's read loop, which delivers what the kernel holds whatever the size of the pending 64 KB view. - the hold from 10 scheduling contexts (top level,
setTimeout,setImmediate, a promise reaction,queueMicrotask,process.nextTick,beforeExit,unhandledRejection, another child'sexit, after anawait): the parent waits in all 10, as on Node. 1.4.3 leaves in all 10. It also waits when theunhandledRejectionhandler runs after the loop went idle, and undernode:testmock timers. - A pipe of an unref'd child is an armed one-shot poll from
unref()on, like any stream that is being read, so a nested loop tick can drop its event (#33261).
Tests. The block spawn().unref() with piped stdio has 16 rows: six stdio shapes that must hold the parent, stdin only, a full stdout alone, a full stdout with stderr open, two stream call sequences that let the parent exit (unref(), and ref(), unref()), two that hold it (unref(), ref(), and unref(), unref(), ref()), a grandchild that holds the pipe, a child that closes its pipes and lives on (POSIX), and a listener attached after the unref() read has buffered the child's output. Two more rows pin data before exit for a launcher, and one chunk for a child that writes before and after unref(). 11 of the 18 fail on bun 1.4.3, the other 7 are guards. The rows ref(), unref() and unref(), unref(), ref() fail on this branch without the ref flag change. The ref flag commit and the late-listener row answer review findings and were not part of the self-review above. The nested rows run one at a time: each starts a bun that starts a bun.
Fail on the machine used here with and without this change (debug build): child_process.test.ts "extra stdio pipes are not double-closed on GC" and "spawn still works with more than 10240 fds open". Four tests that move 8 MiB through a debug child ("destroy() inside a 'data' listener", "maxBuffer against a fast writer") cross the 5 s limit when the machine is busy and pass alone. Upstream: test-child-process-emfile.js in parallel and in sequential (under ulimit -n 64 a debug build cannot open its JS modules on disk, bun 1.4.3 passes both), test-child-process-windows-hide.js, and test-child-process-bad-stdio.js (it needs the test runner). The last two fail on bun 1.4.3 too. Windows and macOS were not run locally.
no test proof · iteration 12 · platform-specific test(s) that do not run on this machine, deferring to CI, which covers all platforms: test/js/node/child_process/child_process.test.ts