install: cache file: tarballs under the integrity the lockfile pins
Closed
RO
1mo ago+374+374+012345678901234567890123456789-90-90-01234567890123456789
Problem
- With a
bun.lockpresent, afile:pkg.tgzdependency is installed from whatever the shared install cache holds under@T@<hash of "pkg.tgz">@@@1. Two projects that both depend onfile:pkg.tgzbut ship different tarballs overwrite each other's entry, and the next lockfile install of one project gets the other project's tarball; the same happens to one project switching between branches whosepkg.tgzdiffers.bun installreports1 package installed, and the integrity inbun.lockis never compared against anything. - Cause:
cached_tarball_folder_name_print(src/install/PackageManager/PackageManagerDirectories.rs) hashes the resolution string, and a local tarball's resolution is the path as written in package.json (relative to the project, or to the workspace), which is not an identity across projects. Only extraction verifies the integrity, and a lockfile install only extracts when the entry is missing (PackageInstall::package_missing_from_cache, and the isolated installer's probe inisolated_install.rs).
Fix
- Local tarballs are cached under the integrity recorded for the package:
@T@sha512-<first 16 digest bytes as hex>@@@1(cached_local_tarball_folder_name), so the name equalssha512sum pkg.tgz | cut -c1-32. URL tarballs keep their URL-keyed names. - Extraction names the entry after the integrity it just verified, or computed when the lockfile had none (
ExtractTarball::lockfile_integrity, passed intomove_to_cache_directorylike the GitHubresolvedname), which is also the value written to the lockfile. So an entry is only reused for the bytes the lockfile pins, different tarballs at the same path coexist, and identical tarballs in different projects share one entry. This is the same model as git checkouts, which are cached under their commit rather than under the URL they were requested by. - A lockfile written before bun recorded tarball integrity cannot name an entry: the name is empty, both installers treat that as a cache miss and extract (
package_missing_from_cache, the isolated probe;determine_preinstall_statealready treated an empty name this way). The hoisted installer already recorded a computed integrity before re-running the waiting installs; the isolated installer now receives the extraction result (on_extract_store_installer) and does the same (Installer::record_extracted_integrity), so the entry can be named, the lockfile is re-saved with the integrity, and the next install is a cache hit. Under--frozen-lockfilesuch a lockfile keeps re-extracting the tarball and is left untouched. compute_cache_dir_and_subpath(bun patch,bun patch --commit) takes the package's integrity and reports a missing one instead of diffing or copying against the cache root. In the normal flow it is present, becausebun patchinstalls first and that install records and saves it.Integrity::Tag::name()is shared between the folder name andDisplay; theDisplayoutput is unchanged.- Verified with
test/cli/install/bun-install-tarball-integrity.test.ts,local tarball cache entriesfor both linkers: two projects sharing a cache keep their tarballs apart, a project reinstalling from a lockfile after another build of the tarball was extracted from the same path gets the pinned build, and a lockfile without an integrity installs from the tarball, records the integrity, and is then served from the entry it recorded. All six fail on the unfixed build at the installed-contents assertion. - Also run with the debug build: the rest of that file,
bun-patch.test.ts(includes thefile:tarball patch flow, which exercises the patched..._patch_hash=entries),bun-install-patch.test.ts,bun-lock.test.ts, the tarball tests ofbun-install.test.ts(except the one fetching from the public internet, which fails identically without this change),bun-add.test.tsandisolated-install.test.ts, and thefile:tarball migration tests inmigrate.test.tsandpnpm-lock-v9.test.ts. Manually:--frozen-lockfilewith a lockfile lacking the integrity (correct contents, lockfile untouched),bun patchplus--commitstarting from such a lockfile, workspace-relativefile:tarballs with both linkers, two aliases of one tarball, andbun add ./pkg.tgz.cargo clippy -p bun_installis clean. - Existing caches are unaffected: the old
@T@<path hash>entries are simply no longer looked up for local tarballs; each local tarball is extracted once more on its next install.
Background
- Install cache:
~/.bun/install/cache(orBUN_INSTALL_CACHE_DIR) is shared by every project on the machine. Each package is extracted once into a folder there (name@version@@@1for npm,@G@<commit>for git,@T@...for tarballs) and installs copy, hardlink or clone out of it; a cache hit is a directory probe, and the tarball bytes are verified only when the folder is created.@@@1is the cache layout version. - Tarball integrity: for
file:and URL tarballs bun hashes the tarball on first extraction and storessha512-...as the third element of the package'sbun.lockentry (["pkg@pkg.tgz", {}, "sha512-..."]); later extractions are verified against it. Lockfiles written before this was recorded, and some migrated lockfiles, have no integrity for these packages. - Install phase callbacks: when a package has to be extracted during the install phase, the installs waiting on it are queued on the extraction task and re-run when it completes (
PackageInstaller::install_enqueued_packages_after_extractionfor the hoisted linker,Installer::on_package_extractedfor the isolated one). The extraction result (ExtractData) carries the integrity that was verified or computed. - Isolated installer threading: its per-package tasks run on the thread pool holding
&Lockfileviews, so the lockfile is only ever written through narrowed raw pointers from the main thread.record_extracted_integritywrites one row through the column's raw pointer (items_raw) before any task of that package starts.
Reproduction (no network)
export BUN_INSTALL_CACHE_DIR=/tmp/tgzcache
for v in one two; do
mkdir -p $v/src/package
echo '{"name":"pkg","version":"1.0.0"}' > $v/src/package/package.json
echo "module.exports = '$v'" > $v/src/package/index.js
tar -C $v/src -czf $v/pkg.tgz package
echo '{"name":"app","dependencies":{"pkg":"file:pkg.tgz"}}' > $v/package.json
done
(cd one && bun install && bun -e 'console.log(require("pkg"))') # one
(cd two && bun install && bun -e 'console.log(require("pkg"))') # two
(cd one && rm -rf node_modules && bun install && bun -e 'console.log(require("pkg"))')
Before: the last line prints two (the cache holds one @T@85f77fa09e18a07f@@@1, last written by project two). After: one, and the cache holds one @T@sha512-...@@@1 entry per tarball.