Hardening round 11: input validation, bounds checks, lifetimes

Merged
JA
4mo ago
+7970-932

What

A hardening and robustness pass across the runtime: input validation, bounds checking, protocol-state handling, and object-lifetime correctness in ~200 files. It contains 122 individual fixes and ~190 new tests (171 new test/it blocks, several parameterized, across 79 existing test files). No new API is introduced; every behavioral change below has a test unless explicitly noted, and each is aligned with Node, the relevant RFC/spec, or the upstream reference implementation.

Potentially breaking / behavior-visible changes

Read this section first. Everything else in the PR preserves behavior for valid inputs.

  • Bun.serve request.url is only synthesized from a structurally valid Host. For every HTTP/1.x request (Bun.serve and node-compat servers alike), a Host value that is empty or contains bytes outside uri-host [":" port] (RFC 3986 authority: alphanumerics, .-:_~%[] and sub-delims) is never used as the authority of the synthesized request.url; request.url falls back to the request target (e.g. /path) and the request is still served. No request is rejected on the basis of the Host field value, and a valid Host still round-trips into request.url exactly. Why: request.url should never carry an authority that cannot come back out of new URL().
  • fetch rejects request lines it cannot legally serialize. A URL path/host (or, for proxied requests, the full href) containing a control character, space, or DEL now fails with InvalidURL before any bytes are written (RFC 9112 request-line grammar; normal fetch() input is percent-encoded by the URL parser and unaffected). Also: a redirect whose Location resolves to a non-http(s) scheme now fails with UnsupportedRedirectProtocol (Fetch spec, matches undici); a 101 arriving on the pre-tunnel leg of a proxied request is treated as an unrequested upgrade; connections whose identity was accepted by a per-request checkServerIdentity callback are never entered into or taken from the keep-alive pool.
  • An own __proto__ key from data files and macros is printed as a computed key. The json, jsonc, json5, toml, yaml, and CSS-module loaders — and objects returned from Bun macros — now emit ["__proto__"]: ... so importing such data yields an own __proto__ property (like JSON.parse) instead of a prototype assignment. Who notices: only code importing data with a __proto__ key; matches esbuild's JSON loader and Node semantics.
  • node:url legacy url.parse lookup tables no longer inherit from Object.prototype (both node:url and the browser fallback), the hostless/slashed lookups use the lowercased protocol, and url.parse(s, true).query is a null-prototype object (empty query included). All three match Node's lib/url.js exactly. Who notices: code doing query.hasOwnProperty(...) or parsing schemes named like toString:.
  • WebSocket client: missing negotiated subprotocol fails the handshake. Per RFC 6455 §4.1, if new WebSocket(url, ["a"]) requested subprotocols and the server's 101 omits Sec-WebSocket-Protocol, the connection now closes with 1002 instead of opening with ws.protocol === "". Matches browsers, ws, and undici. Connections that request no subprotocol are unaffected.
  • HTTP/2 (client and server) enforces RFC 9113 message framing. Trailer blocks must carry END_STREAM and no pseudo-headers; content-length must be 1*DIGIT, non-duplicated, and equal to the DATA actually received (CONNECT exempt) — violations get RST_STREAM(PROTOCOL_ERROR) instead of being delivered. With maxSessionMemory exceeded, new peer streams are refused with REFUSED_STREAM (retryable), and reset streams promptly release their native state — Node/nghttp2 parity throughout. The all-streams teardown helper now throws a TypeError for a non-numeric error code instead of coercing it per stream.
  • node:http2 HTTP/1 fallback (allowHTTP1) frames responses like Node. Header-name matching is case-insensitive; HEAD and close-delimited responses don't get an auto Transfer-Encoding: chunked/terminating chunk; writeHead now throws ERR_HTTP_INVALID_STATUS_CODE / ERR_INVALID_CHAR like Node's ServerResponse. Re-entrant sendTrailers() raises ERR_HTTP2_TRAILERS_ALREADY_SENT in the same order Node does.
  • node:http(s) proxy CONNECT endpoint is validated with validateHeaderValue in release builds (previously a debug-only assertion), so an invalid host/port surfaces as the same error Node throws.
  • Glob: walking through a self-referential directory symlink completes. With followSymlinks, a link that resolves to one of its own live ancestors is descended exactly once (like find -L, glibc fts, node-glob); sibling/cousin links to the same target are still all visited. One pre-existing test changed: it previously asserted the walk failed with ENAMETOOLONG after the path grew past the limit; it now asserts the scan completes.
  • Resolver: an exports/imports target whose expansion would exceed the OS path limit is a normal resolution error (Invalid module specifier / Invalid package target, as Node models it) instead of a hard failure.
  • Shell: template arrays nested deeper than 100 levels throw a clear error instead of recursing without bound; an interpolated string equal to if/then/elif/else/fi is treated as data, never as a reserved word (POSIX: reserved words are only recognized literally); $.escape now quotes strings containing tab, CR, or ? (word delimiters / glob metacharacters).
  • bun pack / bun publish include/exclude matches npm-packlist. With a "files" field, the non-overridable defaults (.git, .npmrc, node_modules, lockfiles) are now applied inside the files traversal too; conversely .hg moved to the overridable default-ignore list, so "files" can re-include it — exactly npm's split.
  • bun upgrade verifies the downloaded artifact against the digest the GitHub Releases API reports for that asset, and fails with a retryable error on mismatch. If the API reports no (or an unrecognized) digest, behavior is unchanged.
  • install: an integrity string carrying several space-separated digests (legal SSRI) is now parsed correctly and verified against the strongest algorithm present (see Deviations); a stored bun.lockb with a non-0/1 byte in a boolean slot fails validation instead of being reinterpreted; lifecycle scripts for registry packages always come from the installed package.json (never from lockfile bytes), matching what Bun writes and what npm does; isolated installs apply the same name/alias shape validation as hoisted installs; bin links reached through a subdirectory get the same resolved-containment check the dotted forms already had (npm only links files inside the package folder).
  • node:fs: mode arguments are no longer masked to 0o777 (setuid/setgid/sticky pass through to the syscall, like Node); copyFile/cp create the destination with the source's permission bits (libuv parity); on Windows, cp copies directory junctions/symlinks via the unprivileged-create + junction-fallback helpers and rewrites \\?\UNC\ targets to \\server\share form (libuv parity), so copying a tree with junctions works without elevation; on macOS the clonefile/openat paths use NOFOLLOW so the copy matches the lstat classification (dereference:false).
  • Web plumbing observable from JS: record conversion (new Headers(obj), fetch init, URLSearchParams, …) snapshots the key list once and re-resolves keys mutated by a converter, exactly as Web IDL specifies (deleted keys skipped, replaced values re-read) — released Bun/Node/WebKit order preserved; TextDecoder.decode over a SharedArrayBuffer or resizable buffer view snapshots the bytes first; consuming a Blob/Response body no longer empties other objects sharing the same byte store (transfer only when sole owner); deeply nested serialized arrays in structuredClone data hit the same recursion cap objects already had; the SIMD decodeURIComponent fast path decodes non-ASCII input as UTF-8 (with U+FFFD for ill-formed sequences) instead of throwing/garbling.
  • N-API / V8 API: napi_create_arraybuffer returns zeroed memory (Node contract); napi_get_typedarray_info/napi_get_dataview_info report the view's real byte_offset; v8::String::Utf8Length returns the exact byte count WriteUtf8 will produce for ill-formed UTF-16; v8::Number::New canonicalizes NaN payloads.
  • Dev-only endpoints check Host/Origin: the inspector (bun --inspect) HTTP/WebSocket endpoint applies its Host/Origin checks before the /json* discovery routes and rejects non-matching DNS-name Host values with 400 (Node inspector semantics); the bake/dev-server internal routes require an allowed Host and same-origin for the error-report/sourcemap endpoints; internal HMR pub/sub topics are namespaced so user publish/subscribe topic strings can never collide with them.
  • markdown: reference-link expansion is charged against md4c's exact output budget (16 × min(input, 64 KiB) scale); once exhausted, further references degrade to literal bracketed text — no error — exactly as md4c does.
  • Misc small behavior corrections: checkPrime validates the candidate before the options (Node's order); HKDF rejects non-secret KeyObjects with ERR_CRYPTO_INVALID_KEY_OBJECT_TYPE (current Node); Ed25519 sign/verify with a wrong-length key errors/returns false; X25519 JWK import honors kty/crv/use/key_ops/ext; SPKAC helpers return false/empty for empty or whitespace-only input (Node); CookieMap.delete of a __Host-/__Secure- cookie emits Secure so browsers accept the expiry; postgres escapeIdentifier rejects embedded NUL (pg parity); valkey/redis pending commands reject with "Connection closed" on disconnect and out-of-band push frames never consume an unrelated command's promise (RESP3); semver strings consisting only of v/=/whitespace parse as * (node-semver); Bun.wrapAnsi measures rows whose seam joins grapheme clusters (combining marks/ZWJ/VS16) the way npm wrap-ansi does; bash/zsh completions handle script names containing : and other special characters; the Docker images verify (not just decode) the release checksum signature.

Changes by area

  • HTTP server (uWS / Bun.serve / node:http) — packages/bun-uws/HttpParser.h, packages/bun-usockets, src/runtime/webcore/Request.rs, src/runtime/server: the request.url Host handling above (URL synthesis in Request.rs only — the HTTP parser's Host handling is unchanged and every request is served); CONNECT requests are framed as an opaque tunnel regardless of Transfer-Encoding/Content-Length (RFC 9110 §9.3.6); TLS socket relocation updates the loop's spill/last-error owner pointers so bookkeeping never points at a moved socket; the bake-only route additionally requires an allowed Host.
  • fetch / HTTP client / webcore — src/http/lib.rs, src/http/ssl_config.rs, src/runtime/webcore/{Request,Blob,TextDecoder}.rs, src/jsc/bindings/webcore/*, src/jsc/bindings/decodeURIComponentSIMD.cpp, src/url/lib.rs, src/runtime/api/BunObject.rs: everything in the highlights, plus: write_request failures propagate their real error instead of being reported as out-of-memory; partial-header/1xx short reads no longer re-feed already-consumed bytes to the parser; SSLConfig now actually applies secureOptions and the client-renegotiation limit/window it was already accepting; URLSearchParams no longer drops a pair whose value has a malformed percent sequence (WHATWG: never drop, decode lazily); AbortSignal native listeners deregistered by an earlier abort callback are not invoked with a stale context; the compression helpers (Bun.gzipSync et al.) read the options object before coercing the input buffer (so a getter can't invalidate the captured slice) and no longer register a deallocator for an empty result's dangling sentinel pointer (an invalid free at GC time under debug allocators).
  • node compat — src/runtime/node/*, src/js/node/*, src/node-fallbacks/url.js, src/jsc/ipc.rs, src/runtime/socket: everything in the highlights, plus: fs path arguments from typed arrays are always pinned for the call's duration; BlockList structured-clone carries only an opaque per-instance nonce resolved through a live table (round-trip unchanged); IPC advanced-mode frame lengths are range-checked before span arithmetic, serialization failures leave no partial frame in the send queue, and a received fd is closed if its message fails to parse; TLS-upgrade initialData is copied to an owned buffer before use; node:wasi interprets rights bitfields as unsigned u64 (per the ABI) and no longer reports success for a path_open that threw internally; the inspector/debugger endpoint changes above.
  • HTTP/2 & WebSocket client — src/runtime/api/bun/h2/connection.rs, h2_frame_parser.rs, src/http_jsc/*: everything in the highlights, plus: 1xx interim responses are not misclassified as trailers; refused streams still HPACK-decode the discarded block (RFC 9113 §4.3) and advance last_stream_id so pipelined RST_STREAMs don't become connection errors; the frame parser no longer holds an exclusive stream reference across calls back into JS (options/header getters, toString coercions) — engine-side stream eviction is deferred until the dispatch unwinds; the trailers failure path still ends the stream with FRAME_SIZE_ERROR + graceful GOAWAY so in-flight streams stay retryable; the deflate plumbing gains a per-VM slot (no behavior change yet).
  • install / pack / bunx / upgrade / create — src/install/*, src/runtime/cli/{pack,create,upgrade}_command.rs, src/semver, packages/bun-release, packages/bun-vscode: everything in the highlights, plus: SSRI option suffixes (?...) are stripped from digest payloads; a GitHub dependency whose resolved ref would not form a single well-formed folder name is refused with a clear error (real refs/SHAs always pass); the trusted-dependency lookup moved off the extraction worker thread (no user-visible change); bun create's package.json rewrite uses the CLI arena so the parsed AST outlives its uses; the npm installer package validates archive entry paths stay inside the destination; the VS Code lockfile preview escapes interpolated text and the debug adapter's session id comes from crypto.randomBytes.
  • resolver / bundler / parsers / macros / sourcemap / markdown — src/resolver, src/bundler, src/parsers/{json,json5,yaml}.rs, src/ast/e.rs, src/js_parser/lexer.rs, src/js_parser_jsc/Macro.rs, src/jsc/RuntimeTranspilerStore.rs, src/jsc/bindings/BunPlugin.cpp, src/sourcemap, src/md, src/paths, src/standalone_graph: the __proto__, resolver-limit, and markdown items above, plus: a native-plugin onLoad source buffer now has exactly one owner (its free callback was registered twice); onResolve callback lists are snapshotted (GC-visible) before user callbacks run, so a callback registering more plugins can't perturb the in-progress dispatch; barrel-import scheduling copies its alias seeds instead of holding references into a map the BFS mutates; embedded bytecode caches are handed to ResolvedSource as a genuinely owned allocation; the lazy sourcemap decompression cache became OnceLock-based (shared-reference safe); the lexer's SIMD long-string fast path advances past scanned bytes (removes a quadratic re-scan on unterminated literals); is_parent_or_equal uses a true prefix check instead of substring containment.
  • shell / glob / CLI / wrapAnsi — src/shell_parser, src/runtime/shell, src/glob/GlobWalker.rs, completions/bun.{bash,zsh}, src/jsc/bindings/wrapAnsi.cpp: the highlights above; the glob walker's followed-link tracking is scoped to the live ancestor chain (DAG revisits still enumerate); wrapAnsi also caches row widths so the seam fix comes with fewer full-row rescans.
  • crypto — src/jsc/bindings/{ncrypto.cpp,node/crypto/*,webcrypto/*}: the highlights above, plus: the sign/verify job copies signature bytes out of the JS view before the async job runs; ECDH.convertKey and prepareAsymmetricKey capture buffer spans only after argument coercions that can run user JS; deserialized CryptoKeys re-validate that the algorithm matches the key class (and an empty key payload is rejected); an OOM while encoding returns after throwing.
  • sql / valkey / s3 — src/sql, src/sql_jsc, src/js/internal/sql, src/runtime/valkey_jsc, src/valkey, src/s3_signing: the highlights above, plus: postgres CopyData payload length is computed per the wire protocol (was one byte short) and PortalSuspended/Copy* messages are consumed instead of desynchronizing the stream; MySQL zero-length AuthSwitchRequest/LocalInfileRequest packets are clean protocol errors instead of a length underflow; prepared-statement caches are keyed by the full statement name, not a 64-bit hash (a hash hit is verified by equality); the distributed-transaction name is type-checked; the S3 region used to synthesize a host must be host-safe and endpoint parsing is index-safe on odd endpoint strings.
  • JSC bindings / N-API / V8 / sqlite / misc — src/jsc/bindings/{napi.cpp,v8/*,ZigException.cpp,ZigGlobalObject.cpp,CookieMap.cpp,sqlite/JSSQLStatement.cpp}, src/jsc/rare_data.rs, src/runtime/webview: the N-API/V8 items above; stack-trace population skips out-of-range frame indices instead of asserting; the native microtask trampoline is hidden from stack traces; bun:sqlite detects a database closed re-entrantly from inside a bind coercion and throws "Database has closed"; the macOS webview bridge type-checks the objects a page posts to its internal message handler.
  • dev server / bake / build & CI — src/runtime/bake/*, dockerhub/*, .github/workflows/update-vendor.yml: the dev-endpoint gating and HMR topic namespacing above; the dev-server terminal error report blanks non-UTF-8 bytes (not just encoded C1); the React SSR flight-data inliner escapes the fully decoded string instead of per-chunk (a boundary-split escape character could previously produce a malformed inline script); Docker images use gpg --verify; the vendor-update workflow passes matrix values through env:.

Deviations / decisions

  • Integrity: strongest-single-digest verification. When an integrity field carries several digests, Bun verifies the strongest supported algorithm present; when several digests of that same algorithm are present, the first is the one verified. npm/ssri accepts a match on any digest of the chosen algorithm — keeping the full set needs plumbing through the manifest cache/lockfile types and is left for a follow-up.
  • No response-decompression size cap in fetch. A per-response decompressed-body limit was implemented and then deliberately removed from this PR ("Keep fetch response decompression unbounded"): Node imposes none, and any cap is a behavior change for legitimate large responses. The net diff has no decompression change.
  • markdown reference expansion degrades instead of erroring, matching md4c exactly (see highlights). No error is ever surfaced.
  • Record conversion keeps the specification's per-property order — [[GetOwnProperty]]/Get interleaved with value conversion, so a toString that mutates a sibling property is observed and a deleted one skipped, the same as released Bun, Node, and WebKit. The property table is never held across user code.
  • Dead code removed because these changes made it unreachable: cache::Entry.external_free_function (plus Entry::new and the free branch of Entry::deinit) and AlreadyBundled::bytecode_slice; the bun_wyhash dependency of sql_jsc is dropped.
  • Deeper, behavior-visible sibling work in the same areas was split into its own PRs so it can be reviewed on its own terms: #33054 (node:https TLS option set), #33055 (websocket dispatch re-entrancy), #33056 (FileSystemRouter/resolver entry locking), #33060 (child_process uid/gid), #33061 (node:http server headers/request timeouts). Nothing from those PRs is claimed here.
  • The changes to CI workflow files, shell completion scripts, Dockerfiles, and the VS Code extension have no automated-test harness; a few lifetime/ordering corrections have no deterministic observation from JS (they are covered by the existing suites and the sanitizer jobs) and are noted as such instead of shipping a non-asserting test.

How it was verified

  • bun bd (full debug build) and cargo check clean with zero warnings; bun run rust:check-all passes 10/10 targets (the change set includes Windows- and macOS-gated code); clippy lints raised on the touched files were addressed.
  • ~190 new regression tests in existing test files (171 test/it blocks across 79 files, several parameterized). Each was verified to fail with USE_SYSTEM_BUN=1 bun test <file> and pass with bun bd test <file> (except the handful noted above with no JS-observable assertion); the touched suites were run in full to confirm no regressions.
  • HTTP/2 changes were exercised against a dedicated h2 conformance suite (test/js/node/http2/h2-conformance.test.ts) and Node's own http2 tests; the node:http Host behavior was checked against Node's conformance test for accepted host values; the record-conversion ordering was checked against WebKit/Node observable order.
  • rustfmt / clang-format / prettier / oxlint clean over the changed files.