install: record a failed bin link or git clone at every log level
Problem
- With the hoisted linker,
bun install --silentexits 0 and saves the lockfile after a bin link fails. Without the flag it exits 1 witherror: Failed to link <name>: ENOTDIR. link_tree_bins(src/install/PackageInstaller.rs:708) adds that error to the log only when the level is not silent. Nothing else records the failure.- A failed git clone has the same check (
src/install/PackageManager/runTasks.rs:1382).
Fix
- Add both errors to the log at every log level. Remove the
log_levelparameter that only this check read. - Correct because the log printer checks the log level, and the install exits 1 on a log error before it saves anything.
- Verified:
test/cli/install/bun-install-registry.test.ts,bun-install-git-deps.test.ts(5 new tests, 3 fail without the fix), plusbun-dedupe,bun-audit,bunx,bun-link.
Background
- The package manager collects errors in
manager.log.install_with_managerprints it unless the level is silent, then exits 1 if it holds an error (install_with_manager.rs:917-924). - Considered
summary.fail, which the other hoisted failures use: that exit comes after the lockfile save. Thedirectories.binread errors are a separate defect, not fixed here.
Downsides
bunx --silentnow exits 1 with no output when itsbun addloses the bin link race of #12917. Of 1,600 concurrent coldbun x --silent node-gypruns, 18 lost it (main: 10, of which 6 still ran the tool).- Bun's node-gyp shim uses that command: 107, 43 and 46 of 500 cold installs of 8, 4 and 2
binding.gyppackages fail (main: 110, 45, 56). - Healthy installs: 209 fewer instructions for 200 bins, same syscalls, same text size.
Notes
Repro. A project depends on a local tarball dep with "bin": {"deptool": "cli.js"}. node_modules/.bin is a regular file, so each bin link fails. Release builds of the merge base and of this branch:
| command (hoisted) | main | this PR |
|---|---|---|
bun install | exit 1, stderr 97 B, no lockfile | the same |
bun install --silent | exit 0, 0 B, lockfile saved | exit 1, 0 B, no lockfile |
bun install --production --silent | exit 1, 0 B, no lockfile | the same |
bun add --silent | exit 0, lockfile saved, package.json changed | exit 1, 0 B, nothing changed |
bun update --silent | exit 0, package.json changed | exit 1, 0 B, nothing changed |
bun remove --silent | exit 1, package.json and lockfile changed | exit 1, 0 B, nothing changed |
healthy bun install --silent | exit 0, 0 B, lockfile saved | the same |
The output of each command without --silent has the same exit code and byte counts on both builds. With a read-only node_modules/.bin (EACCES, not as root) the result is the same as with the regular file.
Git clone. A project with an optional git+file: dependency whose clone fails:
| command | main | this PR |
|---|---|---|
bun install | exit 1, no lockfile | the same |
bun install --silent | exit 1, lockfile saved without the dependency, the next bun install exits 0 | exit 1, no lockfile |
The silent run prints git's own error lines on both builds. This PR adds the line error: InstallFailed cloning repository for <name> (48 B in the probe), which the run without the flag already prints. The log that holds it is printed after the resolve step at every log level, like the errors extracting tarball and checking out repository.
Cost of a healthy install. Release builds of the merge base and of this branch, qemu-x86_64 -one-insn-per-tb -d exec,nochain, 200 local dependencies with one bin each, frozen lockfile, warm cache, 3 runs per build with equal counts:
| main | this PR | |
|---|---|---|
instructions in link_tree_bins | 71,342 | 71,142 |
in link_remaining_bins | 99 | 98 |
in increment_tree_install_count | 8,562 | 8,556 |
in install_hoisted_packages | 6,454 | 6,452 |
symlink / getdents64 calls | 201 / 402 | 201 / 402 |
openat calls (3 runs) | 1224, 1225, 1224 | 1223, 1225, 1224 |
size text of bun | 89,151,860 B | 89,151,860 B |
link_tree_bins / link_remaining_bins | 5,094 B / 1,580 B | 5,113 B / 1,548 B |
The bin link race. Concurrent installs into one folder can fail a bin link with EEXIST (#12917). Bun's node-gyp shim runs bun x --silent node-gyp, and each of those runs bun add node-gyp@latest --silent in one shared folder. On main a process that loses the race exits 0, because the silent run dropped the record. It then runs the tool through the link of the winner, or fails with could not determine executable to run. With this PR it exits 1 with no output.
- 8 concurrent
bun x --silent node-gyp rebuildon a cold folder, 200 times per build (1,600 processes). main: 10 processes lost their own bin link, 6 of them exited 0 and ran the tool, 4 failed withcould not determine executable to run. This PR: 18 lost their own bin link, and all 18 exited nonzero with no output. - Cold
bun installof K packages that have abinding.gyp, 500 per build, batches of 100 that alternate between the builds:
| K | failed installs, main | failed installs, this PR | runs where a nested bun add lost its link, main (install still passed) | the same, this PR |
|---|---|---|---|---|
| 8 | 110 | 107 | 15 (5) | 10 (0) |
| 4 | 45 | 43 | 13 (4) | 13 (0) |
| 2 | 56 | 46 | 35 (0) | 36 (0) |
The race is not changed here. #37851 and #39689 work on it.
Not changed.
- A failed read of a
directories.binfolder is still taken for the end of the folder (src/install/bin.rs:1802,:1964), andbun unlinkdoes not read the result of the bin linker. bunx --silentexits with the code of itsbun addand prints no message.- The security scanner's partial install reads only
summary.fail(security_scanner.rs:85-143), so it does not see a failed bin link. bun install <pkg>,bun link <pkg>andbun update -ido not readmanager.any_failed_to_install.- The isolated linker prints its bin link error at every log level.
Other open PRs on these lines. #43384 edits the same call in hoisted_install.rs. #42039 adds a new use of the log_level parameter in link_tree_bins, which this PR removes. #43076 and #43352 change the same git clone arm: they make a failed clone of an optional dependency a warning. The new git test only compares the run with --silent to the run without it, so it holds with either result.
Suites. bun-install-registry.test.ts (285 pass, 5 todo) and bun-install-git-deps.test.ts (10 pass) with the debug build. bun-dedupe, bun-audit, bun-install-native-binlink, frozen-lockfile-pruned and shebang-normalize pass. bunx, bun-link and bun-install-lifecycle-scripts have failures on this machine that do not come from this change: they need bun or node on a fixed PATH, compare the stdout of a debug build, or measure CPU time.
no test proof · iteration 0 · platform-specific test(s) that do not run on this machine, deferring to CI, which covers all platforms: test/cli/install/bun-install-registry.test.ts, test/cli/install/bun-install-git-deps.test.ts