Give borrowed string bytes a lifetime (Utf8Bytes<'a>, EncodedSlice<'a>)

Merged
DY
2mo ago
+4134-6093

What does this PR do?

Follow-up to #40238. Two string helper types still had no lifetime:

  • ZigString — a {ptr, len} view with encoding bits (Latin-1 / UTF-8 / UTF-16). Now EncodedSlice<'a>, same 16-byte layout, 'a = the bytes it was built over. The lifetime-free POD remains only as the BunString union arm.
  • ZigStringSlice — "these characters as UTF-8 bytes": borrowed when already UTF-8/ASCII, a transcoded Vec otherwise, or a view kept alive by a StringImpl ref. Its borrowed variants were raw pointers, so this compiled: 
    let s = value.to_bun_string(global)?.to_utf8();   // temp String dropped here
use(s.slice());                                    // only valid because to_utf8() had taken a ref
    Now Utf8Bytes<'a>: 
    pub enum Utf8Bytes<'a> { Borrowed(&'a [u8]), Owned(Vec<u8>), Shared(String) /* 8-bit ASCII, holds the ref */ }

impl String {
    fn to_utf8(&self) -> Utf8Bytes<'_>;        // borrow (no ref) or transcode; scoped to self
    fn into_utf8(self) -> Utf8Bytes<'static>;  // for storing: moves self's ref into Shared
}
    to_utf8_without_ref / to_utf8_borrowed fold into to_utf8 (the &self borrow is what keeps the impl alive now). StringView<'a>::to_utf8 is 'a-scoped; JSStringView::to_utf8 is scoped to the guard. SliceWithUnderlyingString becomes Utf8WithString { utf8: Option<Vec<u8>>, string: String } (built by String::into_utf8_with_string()): the Vec only when transcoding was needed, the slice derived from string otherwise, replacing a self-referential raw view.

With lifetimes in place the compiler pointed at code that kept a slice past the Rust value it came from. At runtime these were kept valid by the ref the old to_utf8() took (so none is a live use-after-free on main), but that was invisible in the types; they now own their bytes via into_utf8() or keep the source alive explicitly:

  • code-coverage source-URL map (ByteRangeMapping), shell worker argv (vm_args_utf8), SocketConfig.hostname_or_unix, MySQL Value/query string, S3 credential option strings, bake framework config strings, valkey command arguments, hosted-git-info committish, console JSX tag / %s args, Crypto algorithm name
  • path.join/path.resolve: each argument's JSStringView guard was dropped when the loop advanced while its bytes were kept; for a String object argument whose toString returns a fresh string (Bun's is_string() accepts boxed strings, unlike Node), nothing else rooted that string. The guards are now held until the join/resolve runs. path.basename(path, ext) and udpSocket.send/sendMany get the same scoping (their cells were already rooted).
  • fetch("data:…") moved the URL String value while a slice borrowed through it, and ZigStackFrame.source_url was reassigned while its UTF-8 borrow was live — both fine at runtime (the bytes live in the StringImpl, which held a ref), both rejected by the borrow checker now; restructured

Fixes

Renaming the neutral ZigString::init to EncodedSlice::latin1 made these visibly wrong — they built a Latin-1 view over UTF-8 bytes (the same code existed in Zig and on main), so non-ASCII input decoded as mojibake:

  • Bun.Transpiler#scan / scanImports: import specifiers ("./módulo-ü.js" came back as "./módulo-ü.js") — test added
  • bun test --rerun-each=N under a non-ASCII path: the module-registry key was Latin-1-decoded, the entry was never evicted, and the file body ran once instead of N times — test added
  • Bun.cwd, Bun.origin, and Bun.main when the entry is [eval]/[stdin] or not openable; bun:ffi cc compiler error text and non-ASCII symbol names (as the symbols property key and the function's name); the "N errors building <path>" AggregateError message
  • fs.readFile(path, "utf8") (any string encoding) of an empty file embedded in a bun build --compile executable returned ENOMEM (the empty result was built on a Dead string); it returns "" — test added.
  • Zig::getErrorInstance (C++) used the non-copying toString, so an Error created from a temporary Rust buffer via EncodedSlice::to_error_instance kept message pointing at freed memory (auto-install / package-resolution failure messages, the bun:ffi cc() compile-error message). It now copies, like getTypeErrorInstance already did.

Other observable changes:

  • node:http upgrade with options.headers: a Sec-WebSocket-Protocol / Sec-WebSocket-Extensions value containing bytes ≥ 0x80 was transcoded Latin-1→UTF-8 twice on the wire (é became C3 83 C2 A9); it is passed through once now.
  • new Blob([...parts]) where the string parts come from nested arrays or non-literal values: an all-ASCII result now takes the ASCII fast path (main pessimistically marked it as possibly non-ASCII because the old slice type couldn't tell "borrowed with a ref" from "transcoded").
  • If building the specifier/referrer/url properties of a module-resolution or auto-install error itself fails (string too long / OOM), that exception is cleared and the import() promise is rejected with the error as built so far (main set such a property to "").
  • valkey.hset(key, {field: value, …}) / hmset read the field-name bytes after the property iterator that owned them was destroyed; the iterator now outlives the command send.
  • Not JS-visible: error messages built from a temporary Rust buffer (EncodedSlice::utf8(bytes).to_*_error_instance, formatted create_*_error_instance) are copied once; a String message shares its impl and an argument-free ASCII literal is atomized, all through one BunString__toErrorInstance(kind); coded errors with a literal message (global.err(CODE, "…"), throw_invalid_arguments("…")) get an atomized message because String::create_format returns static_ for an argument-free ASCII literal; EncodedSlice::to_owned_slice no longer writes a hidden NUL past len (no caller read it; to_owned_slice_z is the NUL-terminated form); EncodedSlice::slice() no longer truncates at u32::MAX; JSC__JSFunction__getSourceCode returns an owned string instead of a view into the SourceProvider; S3 credential option strings are stored once (main kept both a boxed copy and the source slice, pinning the JS string for the request).

Also done while every site was being touched:

  • Constructors say what they tag: EncodedSlice::{latin1, utf8, utf16, from_bytes} (was init/init_utf8/init_utf16/from_bytes+with_encoding); String::init (an alias of from_bytes/static_) is removed. JSValue::to_slice → JSValue::to_utf8; eql_comptime/has_prefix_comptime → eq_ascii/starts_with_ascii; PathLike<'a>/StringOrBuffer<'a>/PathOrFileDescriptor<'a> arms are String(Utf8WithString) | ThreadsafeString | Utf8(Utf8Bytes<'a>) (PathLike::Bytes(CowSlice) folded into Utf8): a path borrowed for a synchronous syscall carries the lifetime of its bytes instead of an unchecked CowSlice, and to_thread_safe/from_js exist only on the 'static instantiation so a borrowed path cannot reach the work pool.
  • One spelling for "Rust UTF-8 bytes → JS string": bun_string_jsc::create_utf8_for_js(global, bytes) (≈95 sites that used EncodedSlice::…(x).to_js / String::…(x).to_js); ASCII literals use String::static_("..").to_js. Decoding is identical for valid and invalid UTF-8; the only difference is that a result over the maximum JS string length now throws (STRING_TOO_LONG for 8-bit, an out-of-memory error for 16-bit) instead of evaluating to "", and this path does not consult the synthetic allocation limit (setSyntheticAllocationLimitForTesting / BUN_FEATURE_FLAG_SYNTHETIC_MEMORY_LIMIT), so strings between that limit and the real maximum are returned rather than "".
  • One path for "owned UTF-8 Vec → JS string" (String::from_owned_utf8 / owned_utf8_into_js, used by Utf8WithString, Blob.text(), TextDecoder, Build/ResolveMessage toString); owned_utf16_into_js(global, Vec<u16>) is the UTF-16 counterpart (the separate EncodedSlice__toExternalU16 export is removed). EncodedSliceJsc::to_external_value/external return JsResult instead of ZERO with a pending exception.
  • Errors from a message: global.create_*_error_instance(format_args!(..)) everywhere; JSC__create{Error,TypeError,RangeError}, EncodedSlice__to{Error,TypeError,SyntaxError,RangeError}Instance and helpers.h get*ErrorInstance collapse into BunString__toErrorInstance(str, global, BunErrorKind). create_aggregate_error takes format_args! like its siblings.
  • The BunString PODs (Tag, EncodedSlice, StringImpl, String) move from bun_alloc into bun_core::string; the newtype/Deref layering over them is gone.
  • Removed duplicates/dead code the rename exposed: EncodedSliceJsc::with_encoding (= from_bytes), EncodedSlice::static_ (= latin1), set_output_encoding/detect_encoding (= from_bytes), node_os's private copy of the trait, Utf8Bytes::{init_dupe, is_allocated, take_owned_raw}, nine write-only S3CredentialsWithOptions::_*_slice fields, Terminal::_term_name, Response::get_utf8_url, bun_ast_jsc::{msg_to_js, log_to_js, log_to_js_aggregate_error, log_to_js_array} and css_jsc::error_jsc (callers use bun_jsc::LogJsc / create_error_instance), and unused exports EncodedSlice__free/__toAtomicValue/__to16BitValue/__toExternalValueWithCallback, JSC__JSValue__symbolKeyFor/createRangeError/createTypeError/putRecord/createStringArray/hasOwnProperty, JSC__JSGlobalObject__{get,put}CachedObject, JSC__JSObject__{get,put}Direct, WebCore__DOMURL__href_/pathname_, Zig::toJSString, Zig::toString(EncodedSlice, StringPointer).

C++ side renamed to match: struct EncodedSlice, BunStringTag::EncodedSlice/StaticEncodedSlice, Zig::toEncodedSlice, EncodedSlice__* exports (cppbind type map updated). New debug assertions: String::static_ requires ASCII, EncodedSliceJsc::to_js requires a non-global-allocated slice, EncodedSlice__toExternalValue requires a non-UTF-8 tag. No behaviour change intended beyond the fixes above; to_utf8() on a WTF-backed ASCII string is now zero atomics instead of a ref/deref pair.

How did you verify your code works?

New tests: test/bundler/transpiler/transpiler.test.js (scanImports non-ASCII specifiers), test/cli/test/rerun-each.test.ts (non-ASCII path), and test/bundler/compile-asset-bunfs.test.ts (empty embedded file via readFileSync(.., "utf8"), dirent parentPath, ENOENT err.path) — all fail on 1.4. cargo check/clippy on linux, windows-msvc and macOS targets; debug+ASAN build driven through node:path (join/resolve/basename/relative with non-ASCII), fs with string/Buffer/URL paths, Bun.$ interpolation and $.escape, udpSocket.send/sendMany, fetch("data:"), CryptoHasher, Bun.inspect, WebSocket send/publish, module loading, Bun.Transpiler, RedisClient URL parsing; BUN_JSC_validateExceptionChecks=1 clean on the same script. Ran path, fs, shell, udp, inspect, inspect-error, fetch, serve, http2, websocket-server, socket, text-encoder, zlib, postgres-string-leak and valkey-gc test files on debug and release builds; results match main's debug build (remaining local failures — mkdtempSync() empty name EACCES, broken pipe subproc, Blob utf16 gc — reproduce on main).