Bun.file: stop a file's stat size from capping reads of the whole file

Closed
RO
2mo ago
+159-21

Bun.file(p) reads nothing once anything has resolved the file's size, for any file whose stat size is a lie. procfs, sysfs and cgroupfs regular files all report st_size == 0 and still read hundreds of bytes, and exists()-then-read is the canonical usage sequence, so this reads as "procfs is flaky in Bun".

Repro

const p = "/proc/version"; // any procfs / sysfs / cgroup file

await Bun.file(p).text(); // "Linux version ..." (212 bytes) — a fresh blob is fine

const f = Bun.file(p);
await f.exists(); // true, but this caches the stat size (0) onto the blob
await f.text(); // "" — and so does every later read of f

Reading f.size does the same. .text(), .bytes(), .arrayBuffer(), .json() and .stream() are all affected, and three paths are broken on a fresh blob too, because they resolve the size themselves:

// all of these resolve the size before reading
await drain(new Response(Bun.file(p)).body); // 0 bytes, expected 212
await fetch(url, { method: "POST", body: Bun.file(p) }); // sends 16 bytes, expected 212

const form = new FormData();
form.append("f", Bun.file(p)); // the part body is empty

The same conflation widens a slice. Body::to_readable_stream calls resolve_size() unconditionally, and the File arm overwrites a slice's concrete length with max_size - offset:

await Bun.write("/tmp/hello.txt", "hello world");
const f = Bun.file("/tmp/hello.txt");
await drain(new Response(f.slice(0, 5)).body); // "hello world", expected "hello"

Cause

Blob.size means two different things at once: the number of bytes the view spans, and the upper bound on a read of it. For a file-backed blob the first is only ever a stat hint, but resolve_size() caches that hint into size, and every file read path then uses size as its max_length:

  • Blob::do_read_file / do_read_file_internal pass it to ReadFile
  • ReadableStream::from_blob_copy_ref passes it to FileReader::max_size
  • FormDataContext and fetch_impl pass it to node_fs::read_file's max_size

ReadFile compounds it with a "skip reading an empty regular file" fast path that consults the store's cached mode rather than the mode it just fstat'd, so it only fires once something has already statted the file. That is precisely why a fresh Bun.file(p).text() works and a post-exists() one does not.

node_fs's read_file already gets this right, and has since #1220:

// For certain files, the size might be 0 but the file might still have contents.
// https://github.com/oven-sh/bun/issues/1220

Passing max_size: Some(..) there sets has_max_size and disables that tail read, which is how the FormData and fetch upload sites lost the content.

Fix

Separate the two meanings. Blob::size_is_explicit records that size is a bound the caller asked for with slice(); Blob::read_limit() returns MAX_SIZE ("read until EOF") otherwise, and every file read path uses it. resolve_size() / resolved_size() no longer overwrite a caller-supplied bound for file stores, and ReadFile's unsound fast path is gone — an empty regular file now costs one read() returning 0, which is what the fresh-blob path already did.

ReadFileUV (Windows) keeps its equivalent fast path: it tests the mode from the fstat it just performed, and NTFS regular files do not under-report their size.

.size is unchanged and still matches fs.statSync(p).size, including 0 for procfs and for genuinely empty files.

Relationship to #32794

#32794 fixes the slice widening above from the other side, by clamping a concrete size against the store's size in both arms of resolve_size(). For file stores that clamp is against max_size, which is the value that lies — Bun.file("/proc/version").slice(0, 10) would clamp to min(10, 0) == 0. Leaving a caller-supplied bound alone, as this PR does, is the behaviour both want. The two PRs overlap only on the File arm of resolve_size() / resolved_size(); whichever lands second should take that arm from here. Everything else is disjoint.

Verification

test/js/bun/util/bun-file-read.test.ts. The procfs cases are skipIf(!isLinux); the slice, empty-file and regular-file cases run everywhere.

before (released 1.4.0) / after
$ USE_SYSTEM_BUN=1 bun test test/js/bun/util/bun-file-read.test.ts
(fail) reading a procfs file is not capped by its stat size
(fail) streaming a procfs file is not capped by its stat size
(fail) a procfs file appended to FormData carries its contents
(fail) a procfs file uploaded as a fetch body is sent whole
(fail) a sliced Bun.file() keeps its bounds when read as a Response body
(pass) a sliced Bun.file() keeps its bounds when uploaded as a fetch body
(pass) an empty file still reads empty after exists()
(pass) a regular file still reads after exists()
 4 pass, 5 fail

$ bun bd test test/js/bun/util/bun-file-read.test.ts
 9 pass, 0 fail

Also green: blob.test.ts, bun-file.test.ts, bun-file-fd-read.test.ts, empty-file.test.ts, structured-clone-blob-file.test.ts, blob-cow.test.ts, blob-write.test.ts, bun-serve-file.test.ts, fetch-file-upload.test.ts, FormData*.test.ts, body-stream.test.ts, streams.test.js. bun run rust:check-all passes on all 10 targets.