Keep the file size when preallocating a write

RO
18d ago
+150-17

Fixes #44387

Problem

  • Bun.write(Bun.stdout, data) over 1 KiB under bun x.js >> out.log puts NUL bytes before the data: 1996 NULs for a 2000 byte write.
  • WriteFile::run_with_fd (src/runtime/webcore/blob/write_file.rs:470) calls preallocate_file: fallocate with mode 0, which grows the file to the payload length. >> opens the log with O_APPEND, so the kernel writes at the new end.
  • A short copy keeps the grown size as a NUL tail: Bun.write(path, Bun.file(fd)) from a positioned fd, and shell mv across file systems.

Fix

  • preallocate_file (src/sys/lib.rs) passes FALLOC_FL_KEEP_SIZE. The blocks are still reserved. The write that follows sets the size.
  • Two copy fallbacks in src/sys/lib.rs that called fallocate directly now call preallocate_file.
  • The four copy sites that preallocate (those two, the shell mv fallback, the Bun.write kernel-copy loop) release the reserved blocks after a short copy: bun_sys::release_preallocation (an fstat, then ftruncate to the current size) or an ftruncate to the bytes copied.
  • Correct because no caller reads the size that fallocate set. Each writes or copies its bytes right after (see Notes).
  • Verified: four new tests in test/js/bun/io/bun-write.test.js and test/js/bun/shell/commands/mv.test.ts fail on 1.4.4-canary.1+5c6d4e24c and pass here. Notes lists the other suites.

Background

  • fallocate(2) reserves disk blocks so a large write does not fragment. Mode 0 also sets the file size. FALLOC_FL_KEEP_SIZE does not.
  • Considered an O_APPEND check in WriteFile (#41685) and a trim after each copy (#41487, #41677). Each covers one call site of seven. #33355 named FALLOC_FL_KEEP_SIZE as the clean fix.

Downsides

  • A write that fails midway on Bun.write(fd, string) or fs.writeFile to a caller's fd leaves the reserved blocks past EOF allocated until a truncate or unlink. Measured on ext4: a 4 MiB reservation and a 1 MiB write leave st_blocks at 4 MiB. The copy sites trim them now, these two write sites do not.
  • One fstat per cross-device copy fallback (bun_sys and the shell mv), on a path that already does an openat, a copy and a close. Nothing changes off Linux and Android.
Notes

What changed on 2026-10-23 (a3dbe73f61). The branch conflicted with main in src/install/TarballStream.rs, src/libarchive/lib.rs and test/cli/install/bun-install-streaming-extract.test.ts. main (5c6d4e24cd) is merged in, and the three files are resolved to main's side.

  • Since #44968, both tar writers go through EntryWriter (src/libarchive/lib.rs). It takes the length of an entry from the offset that libarchive returns with ARCHIVE_EOF, it returns an ftruncate failure, it marks the file sparse on Windows, and it does not preallocate.
  • So the tar writers no longer read a size that fallocate set. The tar-writer changes of the earlier revisions of this PR (d4167acebb and before) and its two tar tests are dropped. main's sparse tar members tests cover the entries that end in a hole.
  • The two earlier downsides about tar entries (the NTFS clusters for a trailing hole on Windows, and an extraction that fails when the resize fails) are no longer effects of this PR.
  • Two tests for the copy sites are added (the cases from #41487 and #41677).
  • The source diff against main is now src/sys/lib.rs (three hunks) and one comment in src/runtime/node/node_fs.rs. It is a subset of the source diff that was self-reviewed at d4167acebb (3 concerns raised, all 3 addressed). The merge added no source lines.

Repro on 1.4.4-canary.1+5c6d4e24c:

printf 'abc\n' > out.log
bun -e 'await Bun.write(Bun.stdout, "x".repeat(1999) + "\n")' >> out.log
wc -c < out.log                 # 4000, expected 2004
tr -cd '\0' < out.log | wc -c   # 1996, expected 0

The same through Bun.file(fd) with fs.openSync(p, "a") needs a payload of 256 KiB or more: a shorter string takes the synchronous main-thread path, which does not preallocate.

The four new tests. On 1.4.4-canary.1+5c6d4e24c (the release build of the main commit that is merged in) all four fail. With a debug build of a3dbe73f61 all four pass.

  • appends to stdout redirected with >> (test/js/bun/io/bun-write.test.js): size 4000, expected 2004.
  • appends to a file descriptor opened with 'a' (same file): size 600000, expected 300004.
  • Bun.write(path, Bun.file(fd)) with a positioned fd does not leave preallocated zeros (same file): size 2162688, expected 65536.
  • across devices, the destination is the bytes copied, not st_size (test/js/bun/shell/commands/mv.test.ts): the destination of a sysfs file is 4096 bytes, expected the 5 bytes of content. The test is skipped where /sys/devices/system/cpu/online does not exist.

A probe with 12 cells. A program prints n bytes, and sh redirects them into a log that holds 4 bytes. Bytes in the log afterwards, and how many are NUL:

1,0002,00070,0001,000,000
Bun.write(Bun.stdout), >>, canary 5c6d4e24c1,004, 0 NUL4,000, 1,996 NUL140,000, 69,996 NUL2,000,000, 999,996 NUL
Bun.write(Bun.stdout), >>, this branch1,004, 0 NUL2,004, 0 NUL70,004, 0 NUL1,000,004, 0 NUL

The controls (Bun.write(Bun.stdout) with >, and process.stdout.write with >>) are right on both builds in all four sizes.

Other suites run with the debug build of a3dbe73f61:

  • All of test/js/bun/shell/commands/mv.test.ts: 16 pass, 1 skip, 0 fail.
  • writeFile with a preallocate-sized buffer in test/js/node/fs/fs.test.ts: 4 pass.
  • The sparse tests in test/js/bun/archive.test.ts: 12 pass.
  • sparse tar members in test/cli/install/bun-install-streaming-extract.test.ts: 2 pass.
  • All of test/js/bun/io/bun-write.test.js: 109 pass, 1 fail. The failure is Bun.file(0) survives GC, which hit the 5 s limit in the concurrent run (the file is describe.concurrent, and the debug build has ASAN). It passes alone in 1.0 s. It does not reach a preallocation.
  • cargo check -p bun_sys for aarch64-apple-darwin, x86_64-unknown-freebsd, x86_64-pc-windows-msvc, aarch64-linux-android and x86_64-unknown-linux-musl, and cargo clippy -p bun_sys: all clean.

Syscalls per call, counted from the diff: unchanged. Each site makes one fallocate call, as before.

Per-caller review of preallocate_file with FALLOC_FL_KEEP_SIZE, against main 5c6d4e24cd (seven sites):

  • blob/write_file.rs (Bun.write(dest, string | bytes)): writes the whole payload from the fd position, or appends. The size follows the write. This is the bug of #44387.
  • runtime/node/node_fs.rs (fs.writeFile): writes the buffer, then ftruncate to the written length for a path with a truncating flag. For a caller's fd there is no ftruncate, so a short write no longer leaves a NUL tail either. The O_APPEND skip stays: an append lands at the end of the file, so a reservation at the cursor is not used. Against main this file differs by one comment.
  • blob/copy_file.rs (Bun.write(path, Bun.file(src))): the destination is opened O_TRUNC. copy_file_range copies from the source fd's position and stops at its EOF. The size is what was copied. This is the case from #41487, and the positioned fd test covers it.
  • sys/lib.rs move_file_z_with_handle and copy_file_z_slow_with_handle (cross-device fallbacks, two sites): O_TRUNC destination, then copy_file. The size is what was copied. These two called fallocate directly and now call preallocate_file.
  • shell/builtin/mv.rs (cross-device fallback): O_TRUNC destination, bun_sys::copy_file. This is the case from #41677, and the mv test covers it.
  • jsc/RuntimeTranspilerCache.rs: pwritev of the whole content from offset 0 into a fresh tmpfile.
  • The tar writers (libarchive/lib.rs, install/TarballStream.rs) do not preallocate on main any more.

Reserved blocks after a short write, measured with raw syscalls on ext4 (measured again on 2026-10-23): fallocate(KEEP_SIZE, 0, 4 MiB), write 1 MiB, st_blocks 4 MiB. After ftruncate to 1 MiB: st_blocks 1 MiB. Close alone does not free them. With mode 0, the same short write left 3 MiB of visible NUL bytes in the file. preallocate_file does nothing off Linux and Android, so the change has no effect there.

Related PRs:

  • #41685 (open) adds an O_APPEND check in write_file.rs (superseded by this change) and a fallback_read_write path for copy_file_range EBADF on an O_APPEND destination (independent of fallocate, not addressed here).
  • #41487 and #41677 (closed in favor of this PR) add an ftruncate after a short copy. Their NUL-tail symptom is fixed here, and f5fcab65bd carries their trim: the Bun.write(path, Bun.file(fd)) test with a positioned fd asserts st_blocks under 1 MiB after a 64 KiB copy into a 2 MiB reservation.
  • #36542 (closed): tar extraction with no disk allocation from the entry header and the file length from libarchive. main has that behaviour in EntryWriter, which #44968 added.

no test proof · iteration 1 · platform-specific test(s) that do not run on this machine, deferring to CI, which covers all platforms: test/js/bun/shell/commands/mv.test.ts, test/js/bun/io/bun-write.test.js