#45486 fetch, streams: drop one leading UTF-8 BOM in text() and json() of string and stream bodies

Repro. The bytes are EF BB BF and {"ok":true}.

import { Readable } from "node:stream";
const bytes = () => new Uint8Array([0xef, 0xbb, 0xbf, ...Buffer.from('{"ok":true}')]);
const carriers = [
  "\uFEFF" + '{"ok":true}',
  (async function* () { yield bytes(); })(),
  { async *[Symbol.asyncIterator]() { yield bytes(); } },
  Readable.from([bytes()]),
  new ReadableStream({ type: "direct", pull(c) { c.write(bytes()); c.close(); } }),
];
for (const body of carriers) console.log(await new Response(body).json().catch(e => e.name));
// main: SyntaxError x5.  This PR and node v26.3.0 (first four): { ok: true }

Found by a differential run against Node. No user issue reports these carriers. The one report in this area is #4876 (a fetched byte body), which #7391 answered.

The rule. text() equals new TextDecoder().decode(bytes of the same body), and json() parses that text. Release builds, linux x64, main f14a5a684f against this branch:

The 12 cells are textStream() of a default stream with string chunks. It rejects with TypeError on both builds and is not a BOM matter.

Readers that change: Response / Request text() and json() over a string, an async iterable, a node Readable or a direct stream, textStream() of a string body, ReadableStream.prototype.text() / json() of a direct stream, Bun.readableStreamToText / Bun.readableStreamToJSON of a direct stream, and node:stream/consumers text() / json() of a direct web stream. A marked string body already gave the right text after clone(), .body or new Request(request). It now gives it before them too.

Behavior changes for string chunks of a stream. String chunks are a Bun extension (Node throws TypeError). Two rows of streams.test.js from the C++ streams rewrite (#33193) pinned the old text. They move:

Both extra strips came from 70e57a7da0: a strip in the string arm, a second strip after the default-stream sink, and a strip of the trailing string run wherever it was. A program that reads text() and serves it again no longer passes a leading mark through. A test that asserts a leading U+FEFF from an in-process res.text() of these carriers now fails.

Not covered.

• Lone surrogates. A string body and a string chunk keep them, with or without a mark. bytes() gives U+FFFD. A surrogate pair split across two string chunks is joined. The tests hold rows named exception: for them. To replace them needs a pass over every 16-bit text that is zero-copy today.
• A Blob body that starts with FF FE is read as UTF-16LE (#35190).
• formData() drops the mark where the Fetch standard keeps it (#33631).
• node:stream/consumers text() of a web stream with string chunks: Node appends a string chunk as it is. 3 of 9 probed rows differ from node v26.3.0 (5 of 9 on main). The new tests compare this reader for byte chunks only.
• A producer that writes the same buffer again before close(): the direct text sink reads byte chunks at the end, as on main.
• Bun.$ .json() of output that starts with a mark.
• The two write functions of the text sinks (writeToTextSink, textAccumulatorWrite) stay two copies.

Open PRs in the same functions. None has the same change. Each needs a rebase when it lands second.

• #44269 replaces the all-strings join of convertChunksToText and keeps the two-mark strip of main. Four of its new test rows pin "abc" for two marks. The one that lands second makes the join skip at most one mark and pins those rows again. This is the one behavior decision a maintainer must make.
• #44181 edits finishTextSink, which this PR removes, and both write functions. Its changes move to finishTextAccumulator.
• #40584 edits appendChunkBytes and the chunk loop of finishTextSink.
• #33631 and #35190 overlap in utf8-bom.test.ts only.

Measurements. Release builds of f14a5a684f and of this branch. perf, valgrind, bloaty and strace are not installed here. Per-function counts are gdb single steps from entry to return, callees included, after 20 warm calls.

• The direct rows are JSDirectStreamController::onClose, because the compiler inlines endTextSink in this build.
• Allocation calls inside the measured call: unmarked string body text() 0 and 0, direct byte body 3 and 3, default single buffer 2 and 2, marked 16-bit string body 0 and 4.
• A whole new Response(directStream).text() call is about 14,000 instructions, so +37 is 0.3%.
• Binary: .text 106,386,211 to 106,383,139 bytes (size). endTextSink (3,087 bytes) and stripTextResultBOM (188) are gone. convertChunksToText goes from 8,150 to 4,575. New: finishTextAccumulator 1,976, utf8Decode 604, textOfString 242.
• Microtask turns until text() settles (string, default stream, direct stream, async generator): 1, 1, 4, 6 on both builds.
• Ill-formed UTF-8: 333,450 byte sequences through a direct stream and a default stream, alone and after a string chunk. The text is the same on both builds and equal to one TextDecoder decode.
• A one-chunk body over a SharedArrayBuffer that a worker rewrites: both builds survive 3 of 3 runs.
• Wall clock, a local form of bench/snippets/webstreams-consumers.mjs with 16-bit and direct rows added, 36 cells, 25 interleaved pairs: the A/A noise floor is 7.3%, the A/B deltas are -7.4% to +16.6% with a median of +0.5%. The three cells at the edge of the floor execute the same instruction count on both builds.
• bench/snippets/react-dom-render.bun.js did not run: mitata is not installed here and the installed react-dom has no Bun server build. A local loop over react-dom/server.browser and new Response(stream).text() executes 114,000 to 130,000 instructions per call on main and 115,000 to 118,000 on this branch.
• A marked string body under an address-space limit: text() of 192M code units under a 3 GB limit rejects with RangeError: Failed to allocate memory. main resolves with the mark kept.

Designs weighed. A review compared six designs before the code was written.

• A check when the body is constructed (#33656, my earlier attempt, closed by a stale sweep): one pass over each 16-bit string body, also for bodies that nothing reads as text, and 1 of 5 carriers.
• A strip in finishTextSink only: the two finishers then give different text for bytes followed by a marked string, and 8 of 25 rows stay wrong.
• Every chunk converted to bytes at write time: correct for each row, but each 16-bit string loses the zero-copy path.
• A UTF-16 validity pass to replace lone surrogates: it adds a pass to each 16-bit text().
• The decoder of textStream() for the byte exits: about 3 times faster for valid non-ASCII bytes, but it reads its input twice. With it, a one-chunk body over a SharedArrayBuffer that a worker rewrites crashed 5 of 5 runs. The review also measured a body with one invalid byte about 5 times slower. I did not repeat that run. This PR keeps the decoder that main uses. #41574 owns the shared-memory problem of that decoder.

Self-review. Ten concerns. Nine are addressed:

1. The byte exits used a decoder that reads its input twice. Now: the decoder of main behind a three-byte skip.
2. A marked string was encoded and decoded again (12 ms per 8M units, and an abort under a memory limit). Now: it drops one code unit.
3. The decoder change was a second change with its own speed profile. Removed.
4. Out-of-memory behavior changed without a statement. Now: the stream exits do not change, and the one new copy reports an error.
5. Comments and the title said more than the code does. Narrowed.
6. The test matrix had no documents with invalid bytes and did not read bytes(). Added.
7. The tests pinned node:stream/consumers for string chunks, where Node differs. Now byte chunks only.
8. The Blob FF FE rule is named as not covered.
9. The body names the open PRs in the same functions and the missing user report.
10. Not done: a restack on top of #44269 before this PR opens. #44269 and #44181 have had no maintainer review since October 2. The section above says what the second to land must change.

Verification. bun bd test (debug, ASAN): utf8-bom.test.ts 165 pass, streams.test.js 700 pass and 2 skip, body.test.ts, blob.test.ts, client-fetch.test.ts, streams-string-limit.test.ts, readablestream-helpers.test.ts, and test/js/node/test/parallel/test-stream-consumers.js. The two changed test files also pass with BUN_JSC_validateExceptionChecks=1. On the release build of main, 68 of the 142 new tests of utf8-bom.test.ts fail, and 4 of the 14 new or changed tests of streams.test.js.