shell: reject a stream, used, failed or S3-backed Response body as stdin
RO
16d ago+401+401+012345678901234567890123456789-2-2-0123456789
Problem
$`cat < ${new Response(stream)}`gives the command an empty stdin and exit code 0. So do an unfinishedfetch()body and a used or failed body. An S3-backed body hangs a spawned command.- The redirect setup calls
use_as_any_blob()(src/runtime/shell/states/Cmd.rs:788) oruse_()(src/runtime/shell/Builtin.rs:713). Both return an emptyBlobfor aLocked,UsedorErrorbody (src/runtime/webcore/Body.rs:1287).
Fix
shell::util::throw_if_body_unavailableruns before both calls. A stream or S3-backed body throws... is not supported as a shell redirect yet, as #46021 does for a bare stream. Used, locked and failed bodies throw whatresponse.blob()throws.- A rejected body stays unread, so the hint works. In-memory and file-backed bodies are not changed.
- Verified:
test/js/bun/shell/bunshell.test.ts, "stdin redirect from a Response" (39 tests, 28 fail without the fix). Self-reviewed: 22 concerns, 16 addressed, 6 existing gaps (Notes).
Background
- A
Responsebody is abody::Value: bytes (Blob,InternalBlob,WTFStringImpl), aBlobthat names a file or an S3 object,Locked(a stream orfetch()still supplies the bytes),Used, orError. - A command is a subprocess or a builtin, each with its own redirect setup.
catis a builtin only on Windows.
Downsides
$`echo hi < ${streamResponse}`printedhibefore. It now rejects: the check runs for every command.cat < ${await fetch(url)}still fails when the body arrives after the headers (17 to 20 of 20 runs at 64 KiB). It now throws.- Each
< ${response}redirect pays one call and at most 4 branches. A release build (--lto=off) gains 944 bytes of text.
Notes
The errors
- Stream body:
A Response with a ReadableStream body is not supported as a shell redirect yet. Buffer it first, e.g. `< ${await response.blob()}`, or use Bun.spawn({ stdin: response }) - S3-backed body:
A Response with an S3-backed body is not supported as a shell redirect yet. To redirect from it, pass its bytes: `< ${await response.bytes()}` - Used body, or a body that
.text()orBun.write()is reading:ERR_BODY_ALREADY_USED,Body already used. - A stream that a reader holds:
ERR_INVALID_STATE,Invalid state: ReadableStream is locked. - A failed body (an aborted
fetch()): the stored error.
Why reject
- No user has reported this. A fuzz run found it. Two earlier pull requests for the symptom, #35314 (throw) and #35551 (buffer
fetch()bodies), were closed as stale, with no judgment on the fix. - How often
wc -c < ${await fetch(url)}gets zero bytes, with a fresh process for each run and a loopback server, 20 runs each. 16 KB withContent-Length: 0 of 20 on 1.3.14 and on 1.4.3-canary. 64 KiB: 17 and 20. 100 KB: 15 and 18. 10 KB chunked: 20 and 20. Bun 1.3.14 behaves the same, so this is not a regression. - Reject, not buffer. To buffer the body, the
Cmdneeds a new state that waits on a promise, with a cancel path for a failed script and for a terminated worker. #35551 and #42788 tried that and were withdrawn. #46021 rejects a bareReadableStreamwhen the template is parsed, with the same kind of message. This pull request does the same for aResponsethat wraps one. If the shell learns to stream a body later, theLockedarm can passStdio::ReadableStream, asStdio::extract_body_valuedoes forBun.spawn, and stop throwing. An error that later becomes support is a compatible change. - The check runs when the command starts, not when the template is parsed, because the body can change in between (a
fetch()body finishes, or the script reads it). #42288 made a throw at that point safe: the promise rejects and the script stops. - The
UsedandErrorarms matchStdio::extract_body_value(src/runtime/api/bun/spawn/stdio.rs:306), whichBun.spawn({ stdin: response })uses. The in-use test (action,promise,on_receive_value, disturbed stream) followsPendingValue::take_blob_from_unread_stream.
S3-backed and file-backed bodies
new Response([s3file])andawait fetch(URL.createObjectURL(s3file))have an S3-backedBlobas the body. It has no bytes until a request reads them. On 1.4.3-canary a spawnedcatnever sees the end of its stdin, and a builtin reads zero bytes. The error namesawait response.bytes(), becauseresponse.blob()returns the S3-backedBlobwithout a read. The shell sends no S3 request.new Response(s3file)is different. It is a 302 redirect to the object and has no body, so the command reads nothing. A test pins that.- A plain S3-backed
Blobvalue (< ${s3file}) is not aResponseand is not changed here. #46673 refuses it. - A file-backed body (
new Response(Bun.file(path))) reaches a spawned command as a path or an fd throughStdio::extract_blob. That is not changed, and a test pins it.
Existing gaps that this pull request does not change
- A builtin reads its stdin
BlobthroughBlob::shared_view()(src/runtime/shell/Builtin.rs:852), which is empty for a file-backedBlob. Socat < ${new Response(Bun.file(path))}prints nothing wherecatis a builtin (Windows). A plain< ${Bun.file(fd)}has the same gap, so the fix belongs in the builtin's reader, for both. - The hand-off of a file-backed
Blobto a spawned command (Stdio::extract_blob) keeps only the path or the fd. A slice gives the command the whole file, a relative path resolves against the shell's working directory, and a missing file is created empty.Bun.spawn({ stdin })does the same. - The builtin's check for a
> ${response}target is inverted (src/runtime/shell/Builtin.rs:700). It refuses a file-backed body, and it accepts an in-memory one, whose output is then dropped. #35318 had the fix and was closed as stale. This pull request changes only the stdin redirect. - After
console.log(response)or a read ofresponse.body, an S3-backedBlobhas size 0, andresponse.bytes()asks S3 for one byte. That is in theBlobsize code, not in the shell. - A
Requestis not accepted as a redirect:handle_template_valuetakesResponseonly, so$`cat < ${request}`throwsInvalid JS object used in shell: [object Request].
Measurements and tests
- Size: release builds with
--lto=offin one build directory, this branch againstsrc/frommain(6292c6539d).sizetext 102,233,636 against 102,232,692 bytes: +944 (.text+560,.rodata+384). Data and bss are equal. The stripped file is 102,885,176 bytes on both sides. The check itself is 506 bytes. - Per call: for a body that passes, the check is a variant test, the
match, and for aBlobthe two tests ofis_s3(). It callsto_blob_if_possible()only for aLockedbody, so a string body is not scanned for non-ASCII bytes one more time. - Without the fix (debug build with
src/frommain, and the 1.4.3 canary): 28 of the 39 tests fail. Most fail withresolved with exit code 0 and stdout "". The S3 cases with a spawned command fail on the test timeout, because the command never ends. The 11 that pass are the in-memory, file-backed andnew Response(s3file)controls. - The tests also pass with
BUN_ENABLE_EXPERIMENTAL_SHELL_BUILTINS=1, which makescata builtin on Linux as it is on Windows. - Suites run with the fix:
bunshell.test.ts(503 pass),bunshell-instance,yield,shell-worker-terminate-leak,exec,lazy,throw,bunshell-file. - Docs: the quickstart in
docs/runtime/shell.mdxusedcat < ${await fetch(...)}. It now buffers withawait response.blob(). The redirect section listsBlob(with thebytes()form for anS3File), states what works with every command, and names the bodies that throw. - The branch has
mainmerged in (6292c6539d).
no test proof · iteration 0 · platform-specific test(s) that do not run on this machine, deferring to CI, which covers all platforms: test/js/bun/shell/bunshell.test.ts