spawnSync: keep event-loop ref counts on the loop they were taken on
Problem
bun test --parallelworkers intermittently hang insideBun.spawnSyncof a local bun child: CI printsInterrupted while still running: <file>after 4 minutes, or a run of tests in one worker fails withkilled 1 dangling processandthis test timed out after 90000mswith empty child stdout. The files pass alone. Also reported on macOS (worker at 100% CPU, child a zombie) and reproduced locally in 3 of about 73 CI-shaped batches.- Cause: while spawnSync waits, the VM's loop handle points at spawnSync's private loop, and every keep-alive counter released on whichever loop was current. So a ref taken on the main loop and released during the wait (GC sweeping dead objects, or the runner moving on after a timeout) came off the private loop.
- Once the private loop's count reads 0 it is never polled again, so spawnSync spins without seeing the exit. The private loop is reused, so the worker stays broken for the rest of its life.
- A second bug in the same code: the saved "previous loop handle" was one slot shared by nested spawnSync calls, so after a nested call returned the VM stayed on the private loop for good.
- Fixes #34069
Fix
- Each of the four counters (file polls, KeepAlive refs, the timer and immediate counts, and each JS event loop's keep-alive counter) records the loop it counted on and releases on that same loop, instead of resolving the loop at release time.
- The previous loop handle is returned to the caller and lives on its stack frame, so nested spawnSync calls restore in the right order.
- The property to check: every increment and its matching decrement now hit the same loop. The handle swap itself stays, since spawnSync's own polls rely on it, and each recorded loop outlives what was counted on it (the thread's loop is freed at VM teardown after timers and the heap, the private loop after that).
- Verification: five new test cases fail on the unfixed build (the inner
bun testspins until killed) and pass with the fix; four release one kind of ref each during a wait, and the nesting case alone still fails with only the counter fix applied. Related suites were also run and the remaining failures judged unrelated. The new tests are skipped on Windows, where three of the counters do not go through the swapped handle.
Background
Bun.spawnSyncwaits on a private uws loop with its own epoll/kqueue fd, not the main loop, so main-loop work such as JS timers does not run during the wait. While it waits,vm.event_loop_handleis repointed at that private loop so the child's pidfd and pipe polls register there.- A uws loop tracks how many polls and refs are keeping it alive;
us_loop_run_bun_tickreturns at once without polling when that count is 0. That is why an off-by-one shows up as a busy spin, not a blocked wait. - The four counters the diff touches:
FilePoll(one per fd registered on a loop: pipes, pidfds, sockets),KeepAlive(the on/off ref used by servers, sockets, fetch, fs, dns and similar),timer::All(refs the loop while any JS timer or immediate is pending), andjsc::EventLoop'sconcurrent_ref(refs from MessagePort, BroadcastChannel and similar, folded into a uws loop). - JS runs inside a spawnSync wait through one path: when a bun:test test times out inside spawnSync after its callback has left the stack, the runner continues with the following tests inside the wait. The new tests use that path to release refs during a wait.
no test proof · iteration 0 · Platform-specific test(s) that do not run on this machine. Deferring to CI, which covers all platforms: test/js/bun/spawn/spawnsync-isolated-event-loop.test.ts
Original description
Problem
bun test --parallel workers intermittently stop inside Bun.spawnSync of a local bun child. In CI this is the flaky annotation "failed in the parallel batch ... passed alone": the batch goes quiet, the runner kills it after 4 minutes and prints Interrupted while still running: <file> (24xs), and the file passes in ~100ms on its own. The victims are whatever spawn-heavy files land in the batch (napi do.test.ts files, resolve.test.ts, child_process.test.ts, ctrl-c.test.ts, text-loader.test.ts, malformed-integrity-base64.test.ts, ...), across ubuntu, debian and alpine lanes (for example builds 92307, 92340, 92426, 92160). A second shape of the same failure is a run of consecutive tests in one worker each failing with killed 1 dangling process + this test timed out after 90000ms, with the spawned child's stdout coming back empty (build 92426, resolve.test.ts, six in a row).
#34069 describes the same state outside our CI, on macOS (kqueue): a bun test --parallel (or plain --isolate) worker pegged at 100% CPU inside spawnMaybeSync, the child a zombie, the pipes already closed, tickWithTimeout returning immediately, and the same nested-spawnSync escalation through the bun:test timeout path once the per-test timeout fires inside the spin. The accounting below is shared by the epoll and kqueue backends, and the new tests run on both.
Reproduced locally by running CI-shaped batches (bun test --parallel=3 --timeout=90000, CI's BUN_GARBAGE_COLLECTOR_LEVEL=1 environment, pinned to 4 CPUs): 3 of ~73 batches stalled or timed out the same way, and the stalled worker's main thread was busy-spinning inside BunObject_callback_spawnSync with its child already a zombie.
Cause
spawn_maybe_sync points vm.event_loop_handle at spawnSync's private uws loop for the duration of the call, so that the child's pidfd and pipe polls register on that loop. Everything that maintains a loop's keep-alive bookkeeping resolved that handle at the moment it ran, so it acted on whichever loop happened to be current:
FilePoll::activate/deactivate(num_pollsandactive, plus the epoll/kqueue fd used to register and unregister),KeepAlive::ref_/unref(Loop::ref_/unref), used by servers, sockets, fetch, node:fs, dns, zlib, napi async work, ...timer::All::increment_timer_ref / increment_immediate_ref, which ref the loop when the first JS timer or immediate appears and unref it when the last one goes away,jsc::EventLoop::apply_concurrent_ref_delta, which folds therefKeepAlivecounter used byMessagePort,BroadcastChannelandScriptExecutionContextintovm.platform_loop_opt(), immediately on every JS-thread ref/unref.
So a ref taken on the main loop and released while a spawnSync is waiting was subtracted from the private loop instead. That happens whenever teardown runs during the wait: JSC sweeping a dead Timeout, port or pipe reader while spawnSync allocates its result (very frequent under --isolate + BUN_GARBAGE_COLLECTOR_LEVEL=1, where the previous file's leftovers are swept during the next file), or the test runner carrying on with the following tests after a test times out inside spawnSync. The private loop is reused for every spawnSync in the process, so the error persists for the rest of the worker's life.
us_loop_run_bun_tick returns immediately when loop->num_polls == 0. With the private loop's count off by one, a pipeless spawnSync (the napi harness) starts at 0 and spins from the first tick; a piped one reaches 0 as soon as its pipes hit EOF, before the pidfd event is dispatched. Either way the wait loop spins without polling and never observes the exit. The per-test timeout still fires inside the spin: it prints killed 1 dangling process into the worker's captured output, and if nothing is left to close the spin continues forever (the silent 240s variant); if pipe readers are still open, closing them makes the count non-zero, the loop polls once, and spawnSync returns the long-exited child with empty stdout (the timed-out variant), leaving the count wrong for the next call, hence the consecutive failures. The main loop is left with counts that are too high, which in a plain script also keeps the process from exiting.
A second problem in the same code: SpawnSyncEventLoop kept the saved handle in the struct that both levels of a nested spawnSync share (the timeout path above nests), so after the inner call returned, the outer call restored the private loop's handle and the VM stayed on the private loop for good.
Fix
Each counter records the loop it counted on and uncounts that one:
FilePollgets acounted_loop, used for unregister/deactivate, re-registration and the active-count adjustments (this also makes the unregister hit the epoll/kqueue instance the fd is actually registered with). Because the poll keeps the pointer,FilePoll::register/unregisterand friends now take*mut Loopinstead of&mut Loopand the callers pass the loop handle they already hold; the&'static mutaccessorEventLoopCtx::platform_event_loopgoes away with that.KeepAliverecords the loop it ref'd.unref_on_next_tickkeeps deferring through the pending counter when that is the thread's loop and unrefs directly otherwise, since only the thread's loop drains that counter.timer::Allrecords the loop for each of its two counts.- Every
jsc::EventLooprecords the uws loop it drives (uws_loop, previously a Windows-only field; the VM's loops get the thread's loop inensure_waker, a spawnSync loop gets the private one) and folds its keep-alive counter into that. - The handle returned by
SpawnSyncEventLoop::preparelives on the calling frame and is passed back tocleanup, so nesting restores correctly.
Why this is the right shape: the invariant that broke is "an increment and its matching decrement hit the same loop", and the handle swap itself is what spawnSync's own polls rely on to find the private loop, so the swap stays and each counter remembers its loop instead. That is independent of when or why the release happens (GC, the runner, user code), and the recorded pointers are always live when used: the thread's loop is freed by VirtualMachine::teardown only after timers are cancelled and the heap is destroyed (which is also when the last fold happens), and the private loop, owned by RareData, is freed after that. On Windows the first three counters go through the thread's loop regardless of the handle, so their new fields are cfg(not(windows)); the EventLoop counter did follow the swapped handle there too and is fixed on every platform.
Tests
test/js/bun/spawn/spawnsync-isolated-event-loop.test.ts gets a block of five cases. Each runs a small file under bun test in which test a enters a spawnSync after leaving its synchronous section and times out, so the runner executes the following tests while the private loop is current. a's spawnSync holds exactly one poll on the private loop, so one misdirected release is enough to make it spin forever; on the fixed build the release comes off the main loop, a's (killed) child is observed, and the file finishes with everything but a passing.
- four cases release one ref each of a different kind inside the wait: a JS timer (
clearTimeout), aKeepAlive(Bun.serve().unref()), a registeredFilePoll(cancelling a subprocess's stdout), and an event-loop keep-alive ref (BroadcastChannel.unref()); - the fifth nests a spawnSync inside the wait and then checks that the main loop is polled again afterwards (a child whose pidfd is registered on the main loop has its exit observed).
On the unfixed build all five fail (the inner bun test spins, the outer test times out and bun test kills it). On a build with the counter fixes but without the nested-handle fix, the four release cases pass and only the nesting case fails, so the cases are independent. Each case takes ~1.4s on a debug build; the block is deliberately sequential because bun test only kills a timed-out test's dangling processes when the test is not in a concurrent group, and that is what cleans up a spinning inner process on a regressed build.
Also ran the spawn, spawnSync, maxBuffer, pidfd, child_process, BroadcastChannel, MessagePort/worker, dns, macro, fs.watch, timers and setImmediate suites on the debug build, plus bun test --parallel=3 over the CI victim files with BUN_GARBAGE_COLLECTOR_LEVEL=1. The failures left in those runs are unrelated to this change: network-dependent dns lookups, RSS and throughput tests over their budget under ASAN, a pre-existing LSan report in worker-terminate-lifetime that reproduces with a plain require("fs"), and container-specific process/uid cases; the ones that were checked against an unfixed debug build fail identically there.
Fixes #34069