shell: reject a stream, used, failed or S3-backed Response body as stdin

RO
16d ago
+401-2

Problem

  • $`cat < ${new Response(stream)}` gives the command an empty stdin and exit code 0. So do an unfinished fetch() body and a used or failed body. An S3-backed body hangs a spawned command.
  • The redirect setup calls use_as_any_blob() (src/runtime/shell/states/Cmd.rs:788) or use_() (src/runtime/shell/Builtin.rs:713). Both return an empty Blob for a Locked, Used or Error body (src/runtime/webcore/Body.rs:1287).

Fix

  • shell::util::throw_if_body_unavailable runs before both calls. A stream or S3-backed body throws ... is not supported as a shell redirect yet, as #46021 does for a bare stream. Used, locked and failed bodies throw what response.blob() throws.
  • A rejected body stays unread, so the hint works. In-memory and file-backed bodies are not changed.
  • Verified: test/js/bun/shell/bunshell.test.ts, "stdin redirect from a Response" (39 tests, 28 fail without the fix). Self-reviewed: 22 concerns, 16 addressed, 6 existing gaps (Notes).

Background

  • A Response body is a body::Value: bytes (Blob, InternalBlob, WTFStringImpl), a Blob that names a file or an S3 object, Locked (a stream or fetch() still supplies the bytes), Used, or Error.
  • A command is a subprocess or a builtin, each with its own redirect setup. cat is a builtin only on Windows.

Downsides

  • $`echo hi < ${streamResponse}` printed hi before. It now rejects: the check runs for every command.
  • cat < ${await fetch(url)} still fails when the body arrives after the headers (17 to 20 of 20 runs at 64 KiB). It now throws.
  • Each < ${response} redirect pays one call and at most 4 branches. A release build (--lto=off) gains 944 bytes of text.
Notes

The errors

  • Stream body: A Response with a ReadableStream body is not supported as a shell redirect yet. Buffer it first, e.g. `< ${await response.blob()}`, or use Bun.spawn({ stdin: response })
  • S3-backed body: A Response with an S3-backed body is not supported as a shell redirect yet. To redirect from it, pass its bytes: `< ${await response.bytes()}`
  • Used body, or a body that .text() or Bun.write() is reading: ERR_BODY_ALREADY_USED, Body already used.
  • A stream that a reader holds: ERR_INVALID_STATE, Invalid state: ReadableStream is locked.
  • A failed body (an aborted fetch()): the stored error.

Why reject

  • No user has reported this. A fuzz run found it. Two earlier pull requests for the symptom, #35314 (throw) and #35551 (buffer fetch() bodies), were closed as stale, with no judgment on the fix.
  • How often wc -c < ${await fetch(url)} gets zero bytes, with a fresh process for each run and a loopback server, 20 runs each. 16 KB with Content-Length: 0 of 20 on 1.3.14 and on 1.4.3-canary. 64 KiB: 17 and 20. 100 KB: 15 and 18. 10 KB chunked: 20 and 20. Bun 1.3.14 behaves the same, so this is not a regression.
  • Reject, not buffer. To buffer the body, the Cmd needs a new state that waits on a promise, with a cancel path for a failed script and for a terminated worker. #35551 and #42788 tried that and were withdrawn. #46021 rejects a bare ReadableStream when the template is parsed, with the same kind of message. This pull request does the same for a Response that wraps one. If the shell learns to stream a body later, the Locked arm can pass Stdio::ReadableStream, as Stdio::extract_body_value does for Bun.spawn, and stop throwing. An error that later becomes support is a compatible change.
  • The check runs when the command starts, not when the template is parsed, because the body can change in between (a fetch() body finishes, or the script reads it). #42288 made a throw at that point safe: the promise rejects and the script stops.
  • The Used and Error arms match Stdio::extract_body_value (src/runtime/api/bun/spawn/stdio.rs:306), which Bun.spawn({ stdin: response }) uses. The in-use test (action, promise, on_receive_value, disturbed stream) follows PendingValue::take_blob_from_unread_stream.

S3-backed and file-backed bodies

  • new Response([s3file]) and await fetch(URL.createObjectURL(s3file)) have an S3-backed Blob as the body. It has no bytes until a request reads them. On 1.4.3-canary a spawned cat never sees the end of its stdin, and a builtin reads zero bytes. The error names await response.bytes(), because response.blob() returns the S3-backed Blob without a read. The shell sends no S3 request.
  • new Response(s3file) is different. It is a 302 redirect to the object and has no body, so the command reads nothing. A test pins that.
  • A plain S3-backed Blob value (< ${s3file}) is not a Response and is not changed here. #46673 refuses it.
  • A file-backed body (new Response(Bun.file(path))) reaches a spawned command as a path or an fd through Stdio::extract_blob. That is not changed, and a test pins it.

Existing gaps that this pull request does not change

  • A builtin reads its stdin Blob through Blob::shared_view() (src/runtime/shell/Builtin.rs:852), which is empty for a file-backed Blob. So cat < ${new Response(Bun.file(path))} prints nothing where cat is a builtin (Windows). A plain < ${Bun.file(fd)} has the same gap, so the fix belongs in the builtin's reader, for both.
  • The hand-off of a file-backed Blob to a spawned command (Stdio::extract_blob) keeps only the path or the fd. A slice gives the command the whole file, a relative path resolves against the shell's working directory, and a missing file is created empty. Bun.spawn({ stdin }) does the same.
  • The builtin's check for a > ${response} target is inverted (src/runtime/shell/Builtin.rs:700). It refuses a file-backed body, and it accepts an in-memory one, whose output is then dropped. #35318 had the fix and was closed as stale. This pull request changes only the stdin redirect.
  • After console.log(response) or a read of response.body, an S3-backed Blob has size 0, and response.bytes() asks S3 for one byte. That is in the Blob size code, not in the shell.
  • A Request is not accepted as a redirect: handle_template_value takes Response only, so $`cat < ${request}` throws Invalid JS object used in shell: [object Request].

Measurements and tests

  • Size: release builds with --lto=off in one build directory, this branch against src/ from main (6292c6539d). size text 102,233,636 against 102,232,692 bytes: +944 (.text +560, .rodata +384). Data and bss are equal. The stripped file is 102,885,176 bytes on both sides. The check itself is 506 bytes.
  • Per call: for a body that passes, the check is a variant test, the match, and for a Blob the two tests of is_s3(). It calls to_blob_if_possible() only for a Locked body, so a string body is not scanned for non-ASCII bytes one more time.
  • Without the fix (debug build with src/ from main, and the 1.4.3 canary): 28 of the 39 tests fail. Most fail with resolved with exit code 0 and stdout "". The S3 cases with a spawned command fail on the test timeout, because the command never ends. The 11 that pass are the in-memory, file-backed and new Response(s3file) controls.
  • The tests also pass with BUN_ENABLE_EXPERIMENTAL_SHELL_BUILTINS=1, which makes cat a builtin on Linux as it is on Windows.
  • Suites run with the fix: bunshell.test.ts (503 pass), bunshell-instance, yield, shell-worker-terminate-leak, exec, lazy, throw, bunshell-file.
  • Docs: the quickstart in docs/runtime/shell.mdx used cat < ${await fetch(...)}. It now buffers with await response.blob(). The redirect section lists Blob (with the bytes() form for an S3File), states what works with every command, and names the bodies that throw.
  • The branch has main merged in (6292c6539d).

no test proof · iteration 0 · platform-specific test(s) that do not run on this machine, deferring to CI, which covers all platforms: test/js/bun/shell/bunshell.test.ts