fetch: normalize string bodies so text() matches the UTF-8 decode of arrayBuffer()

Closed
RO
2mo ago
+84-0

What

new Response(str).text() / .json() now match new TextDecoder().decode(await new Response(str).arrayBuffer()) for every string body: a leading U+FEFF BOM is stripped and lone surrogates become U+FFFD, matching Node, browsers, and Bun's own byte-backed body paths.

Repro

const doc = "\ufeff" + '{"a":1}';
await new Response(doc).json();                            // before: SyntaxError: Failed to parse JSON
await new Response(new TextEncoder().encode(doc)).json();  // always: {a: 1}

await new Response("a\uD800b").text();                     // before: "a\uD800b" (lone surrogate kept)
new TextDecoder().decode(await new Response("a\uD800b").arrayBuffer());  // always: "a\uFFFDb"

Node v26 and browsers return {a: 1} / "a\uFFFDb" on every row.

Cause

A string BodyInit is its UTF-8 encoding (fetch "extract a body", USVString), and the Body mixin text() / json() are "UTF-8 decode" of those bytes. UTF-8 decode strips a leading EF BB BF and cannot produce a lone surrogate.

Bun's string-body fast path stores the raw WTFStringImpl (body::Value::from_js, src/runtime/webcore/Body.rs) and .text() hands it straight back (Any::to_string, Any::WTFStringImpl arm) with no encode/decode round trip, so a leading BOM and any lone surrogate survived. .arrayBuffer()/.bytes() of the same body transcode via to_utf8_alloc and do normalize, so the two disagreed.

Fix

In body::Value::from_js, when the incoming JS string is 16-bit and either starts with 0xFEFF or fails simdutf::validate::utf16le (i.e. contains a lone surrogate), encode it to UTF-8 and store as InternalBlob so every consumer agrees. 8-bit Latin-1 strings cannot hold U+FEFF or surrogates, and well-formed 16-bit strings pass the SIMD validator, so both keep the zero-copy WTFStringImpl path.

Verification

test/js/web/fetch/utf8-bom.test.ts gains a describe.each over Response / Request string bodies covering BOM-only, double-BOM, interior BOM preserved, emoji, lone high/low surrogates, and a text() === new TextDecoder().decode(arrayBuffer()) invariant over a dozen string shapes.

bun bd test test/js/web/fetch/utf8-bom.test.ts   # 63 pass / 0 fail
USE_SYSTEM_BUN=1 bun test ...                    # 33 pass / 30 fail

Also ran body.test.ts, response.test.ts, body-mixin-errors.test.ts, body-clone.test.ts (all green) and BUN_JSC_validateExceptionChecks=1.