#43183 shell: fail every queued chunk when an IOWriter dies, not one per child

The pipeline case. The pipe between two stages is a socketpair with an IOWriter on the write end (src/runtime/shell/states/Pipeline.rs). ls -R starts one thread pool task per directory. Each finished task queues its listing as one chunk under the builtin's ChildPtr. The next stage (true, pwd, a second ls, an external command that exits early) closes the read end without reading. The first write fails with EPIPE while several chunks are queued. A debug log of the unfixed build shows four OutputTask starts and one onIOWriterChunk. The hang is racy because it needs two or more chunks in the queue when the write fails. A chunk that is queued after the failure is rejected by handle_dead_writer with its own completion, which is why a single operand never hangs.

Measured, never settled of 10 runs each (ls -R d | ls -R d, ls -R d | pwd, ls -R d | true; the tree has 4 directories and 80 files):

The controls (ls -R d | cat, ls d | ls d, seq 400 | ls -R d) settle in every run on every build.

Why the fix is in the queue and not in the builtins. The OutputTask builtins keep one FIFO for the chunks they queued on stdout and on stderr, and both can be different writers. One notification per child cannot tell them how many of their chunks failed. The Zig shell registered every OutputTask as its own writer child, so one error per child was one error per chunk there. The Rust port put all chunks of a builtin under the builtin's ChildPtr and kept the per-child error path. rm counted chunks under one child in Zig too.

Children of IOWriter, checked one by one. Only cat, rm, ls, mkdir, touch, cp and CapturedWriter can have two or more chunks queued. rm and the OutputTask builtins count completions and ignore the error value. cat and CapturedWriter finish at their first error, so they cancel. Every other child (echo, pwd, which, seq, yes, basename, dirname, cd, exit, export, mv, Cmd, CondExpr, Pipeline) waits for its one chunk before it queues the next.

on_sync_error. The chunk of the child whose enqueue is on the stack is still returned, not dispatched, so the trampoline delivers it after enqueue unwinds. Other queued chunks are dispatched inline, as before. CapturedWriter::do_write only runs while its reader is pending, so an inline completion there cannot free the relay.

The cat change. The file form of the builtin cat set out_done when its output queue drained and never cleared it. When more input arrived later, cat still finished at the end of its input, with the new chunks queued. Builtin::done frees the node, the next command reuses the id, and the completions of the leftover chunks go to that command (or to a freed node). The builtin is the default cat on Windows and is behind BUN_ENABLE_EXPERIMENTAL_SHELL_BUILTINS elsewhere. Measured with cat fifo; echo tail, where the test reads first\n back before it writes 1 MB more to the FIFO:

On Windows, cat big.txt | cat > out.txt (1 MB) five times in one process panics with expected Node::Cmd at Node#3, got Free in 6 of 6 processes on the canary, and in none on this branch. The read-error path of cat (cancel, then suspend) is not changed here. #37743 is open for it.

Tests.

• pipeline stage whose reader exits without reading (new, in-process, runs on Windows too except the two rows with external commands): ls -d, mkdir -v and rm -v with 64 operands into true, ls -R of 300 directories into true, into true | cat, into sh -c 'exit 0', and into head -n 1.
• command output after the stdout reader went away (from #37719, reworked): epipe-fixture.ts waits until its own stdout fails with EPIPE, then runs ls -d, mkdir -v, rm -v or a relayed head -c 1048576 /dev/zero ten times. It has no timed wait. A run takes the path under test only when several chunks are queued before the first write fails. Ten runs replace the 100 ms pause that #37719 used for that.
• On main the 7 pipeline tests and the 3 fixture builtins time out: 3 of 3 runs of the file on a debug ASAN build, 4 of 4 on a release build.
• The relay test guards the CapturedWriter cancel. With that call removed, it fails in 7 of 8 runs on a debug ASAN build (heap-use-after-free in CapturedWriter::on_iowriter_chunk from fail_pending_writers, or a timeout).
• cat whose input ends while its output is still queued: the two FIFO shapes from the table above (Linux, with the experimental builtins flag in a child process), and on Windows cat big.txt | cat > out.txt five times in one child process. The FIFO tests do not run on macOS: both timed out there in CI because the handshake never completed, and I could not find out why without a macOS machine. The cat logic they cover is the same on every platform. The read-everything test fails on main in 3 of 3 runs, the Windows test in 4 of 4.
• cat with several chunks queued (Windows only, where cat is a builtin by default) runs cat big.txt | true with a 1 MB file in a child process. It guards the cat cancel. With that call removed, the child panics with expected Node::Cmd at Node#3, got Free in every run on a Windows debug build. It passes on the canary, which has no per-chunk completions.

Related open PRs. #40228 fixes the same hang in another way: it keeps the one-completion-per-child path (IOWriter.rs:792 and :830 on its head) and gives every OutputTask chunk its own identity (ChildPtr::builtin_task(node, seq)), with closed-pipe tests for ls, mkdir and rm. #42819 rewrites the failure path around a per-chunk failed queue, which is the contract of this PR, and it also calls cancel_chunks in CapturedWriter::on_iowriter_chunk. Both are large refactors (68 and 419 files). This PR is the small version of the fix against main. If it lands first, #40228 no longer needs the dedup or the per-chunk seq for this hang, but the two cancel_chunks calls and the cat count have to carry over: its cat.rs still has out_done. The open design question is which contract the queue keeps: one completion per queued chunk (this PR, #42819) or one per child identity (#40228).

Not changed. After the EPIPE, ls -R still walks the rest of the tree before it finishes. It settles, but a large tree takes as long as a full listing. Exit codes on EPIPE are not changed and not asserted for the failing stage (#40033 changes them).