install: cache file: tarballs under the integrity the lockfile pins

Closed
RO
1mo ago
+374-90

Problem

  • With a bun.lock present, a file:pkg.tgz dependency is installed from whatever the shared install cache holds under @T@<hash of "pkg.tgz">@@@1. Two projects that both depend on file:pkg.tgz but ship different tarballs overwrite each other's entry, and the next lockfile install of one project gets the other project's tarball; the same happens to one project switching between branches whose pkg.tgz differs. bun install reports 1 package installed, and the integrity in bun.lock is never compared against anything.
  • Cause: cached_tarball_folder_name_print (src/install/PackageManager/PackageManagerDirectories.rs) hashes the resolution string, and a local tarball's resolution is the path as written in package.json (relative to the project, or to the workspace), which is not an identity across projects. Only extraction verifies the integrity, and a lockfile install only extracts when the entry is missing (PackageInstall::package_missing_from_cache, and the isolated installer's probe in isolated_install.rs).

Fix

  • Local tarballs are cached under the integrity recorded for the package: @T@sha512-<first 16 digest bytes as hex>@@@1 (cached_local_tarball_folder_name), so the name equals sha512sum pkg.tgz | cut -c1-32. URL tarballs keep their URL-keyed names.
  • Extraction names the entry after the integrity it just verified, or computed when the lockfile had none (ExtractTarball::lockfile_integrity, passed into move_to_cache_directory like the GitHub resolved name), which is also the value written to the lockfile. So an entry is only reused for the bytes the lockfile pins, different tarballs at the same path coexist, and identical tarballs in different projects share one entry. This is the same model as git checkouts, which are cached under their commit rather than under the URL they were requested by.
  • A lockfile written before bun recorded tarball integrity cannot name an entry: the name is empty, both installers treat that as a cache miss and extract (package_missing_from_cache, the isolated probe; determine_preinstall_state already treated an empty name this way). The hoisted installer already recorded a computed integrity before re-running the waiting installs; the isolated installer now receives the extraction result (on_extract_store_installer) and does the same (Installer::record_extracted_integrity), so the entry can be named, the lockfile is re-saved with the integrity, and the next install is a cache hit. Under --frozen-lockfile such a lockfile keeps re-extracting the tarball and is left untouched.
  • compute_cache_dir_and_subpath (bun patch, bun patch --commit) takes the package's integrity and reports a missing one instead of diffing or copying against the cache root. In the normal flow it is present, because bun patch installs first and that install records and saves it.
  • Integrity::Tag::name() is shared between the folder name and Display; the Display output is unchanged.
  • Verified with test/cli/install/bun-install-tarball-integrity.test.ts, local tarball cache entries for both linkers: two projects sharing a cache keep their tarballs apart, a project reinstalling from a lockfile after another build of the tarball was extracted from the same path gets the pinned build, and a lockfile without an integrity installs from the tarball, records the integrity, and is then served from the entry it recorded. All six fail on the unfixed build at the installed-contents assertion.
  • Also run with the debug build: the rest of that file, bun-patch.test.ts (includes the file: tarball patch flow, which exercises the patched ..._patch_hash= entries), bun-install-patch.test.ts, bun-lock.test.ts, the tarball tests of bun-install.test.ts (except the one fetching from the public internet, which fails identically without this change), bun-add.test.ts and isolated-install.test.ts, and the file: tarball migration tests in migrate.test.ts and pnpm-lock-v9.test.ts. Manually: --frozen-lockfile with a lockfile lacking the integrity (correct contents, lockfile untouched), bun patch plus --commit starting from such a lockfile, workspace-relative file: tarballs with both linkers, two aliases of one tarball, and bun add ./pkg.tgz. cargo clippy -p bun_install is clean.
  • Existing caches are unaffected: the old @T@<path hash> entries are simply no longer looked up for local tarballs; each local tarball is extracted once more on its next install.

Background

  • Install cache: ~/.bun/install/cache (or BUN_INSTALL_CACHE_DIR) is shared by every project on the machine. Each package is extracted once into a folder there (name@version@@@1 for npm, @G@<commit> for git, @T@... for tarballs) and installs copy, hardlink or clone out of it; a cache hit is a directory probe, and the tarball bytes are verified only when the folder is created. @@@1 is the cache layout version.
  • Tarball integrity: for file: and URL tarballs bun hashes the tarball on first extraction and stores sha512-... as the third element of the package's bun.lock entry (["pkg@pkg.tgz", {}, "sha512-..."]); later extractions are verified against it. Lockfiles written before this was recorded, and some migrated lockfiles, have no integrity for these packages.
  • Install phase callbacks: when a package has to be extracted during the install phase, the installs waiting on it are queued on the extraction task and re-run when it completes (PackageInstaller::install_enqueued_packages_after_extraction for the hoisted linker, Installer::on_package_extracted for the isolated one). The extraction result (ExtractData) carries the integrity that was verified or computed.
  • Isolated installer threading: its per-package tasks run on the thread pool holding &Lockfile views, so the lockfile is only ever written through narrowed raw pointers from the main thread. record_extracted_integrity writes one row through the column's raw pointer (items_raw) before any task of that package starts.
Reproduction (no network)
export BUN_INSTALL_CACHE_DIR=/tmp/tgzcache
for v in one two; do
  mkdir -p $v/src/package
  echo '{"name":"pkg","version":"1.0.0"}' > $v/src/package/package.json
  echo "module.exports = '$v'" > $v/src/package/index.js
  tar -C $v/src -czf $v/pkg.tgz package
  echo '{"name":"app","dependencies":{"pkg":"file:pkg.tgz"}}' > $v/package.json
done
(cd one && bun install && bun -e 'console.log(require("pkg"))')   # one
(cd two && bun install && bun -e 'console.log(require("pkg"))')   # two
(cd one && rm -rf node_modules && bun install && bun -e 'console.log(require("pkg"))')

Before: the last line prints two (the cache holds one @T@85f77fa09e18a07f@@@1, last written by project two). After: one, and the cache holds one @T@sha512-...@@@1 entry per tarball.