install: derive the bun.lockb meta hash in the writer

Draft
RO
1mo ago
+1047-404

Problem

  • A bun.lockb written from pnpm-lock.yaml stores an all-zero meta hash. The next bun ci exits 1 with error: lockfile had changes, but lockfile is frozen.
  • The writer copies Lockfile.meta_hash (src/install/lockfile/bun.lockb.rs:186). The pnpm migrator never fills that field. The yarn and npm migrators fill it without lifecycle scripts, so a root postinstall fails the same check.

Fix

  • generate_meta_hash takes the script lines from the package columns. The writer hashes the packages it writes, unless the comparison before the save did.
  • Lockfile.meta_hash becomes loaded_meta_hash: Option<MetaHash>, which code only compares. Without a stored hash, the frozen check uses Lockfile::eql.
  • bun pm migrate and bun pm trust save through one entry. It prepares a migrated lockfile like the next install does.
  • Verified: 28 new tests in test/cli/install/bun-lockb.test.ts and test/cli/install/migration/, 20 fail on 1.4.3-canary.1. Notes list the other suites.

Background

  • The meta hash is SHA-512/256 over the sorted name@resolution lines and the lifecycle scripts of the root and workspaces. A frozen install compares the stored hash with a fresh one.
  • bun.lock stores no hash. Its frozen check is Lockfile::eql.
  • A review weighed one line in the pnpm migrator. The field then goes stale after bun add.

Downsides

  • bun pm migrate to bun.lockb does more work. Its save step takes 1.6M instructions and 52 allocations at 202 packages (base 0.15M, 18).
  • bun pm hash-print prints zeros in a project with only yarn.lock or package-lock.json.
  • Installs with bun.lockb pay nothing new: save_to_disk takes 871,346 instructions at 1,002 packages (base 871,273).
Notes

Measurements. Release builds of the merge base and of this branch, linux-x64. Counts come from gdb breakpoints and a ptrace single-step counter. strace, perf, valgrind and bloaty are not in the container, so syscall counts use gdb catchpoints and sizes use size.

  • generate_meta_hash calls per flow (base/PR): bun.lock no-op 0/0, fresh bun.lock 1/0, text --lockfile-only 1/0, bun.lock frozen 0/0, bun.lock add 0/0, bun.lockb no-op 1/1, bun.lockb fresh 1/1, bun.lockb frozen 1/1, bun.lockb add 1/1, bun.lockb remove 1/1, bun.lockb --lockfile-only 1/1, pnpm to bun.lockb install 0/1, pnpm to bun.lock install 0/0, npm to bun.lock install 2/2, npm to bun.lockb install 2/2, yarn to bun.lock install 2/2, bun pm ls on a yarn.lock project 1/0, bun pm ls on a pnpm project 0/0, bun pm migrate pnpm to bun.lockb 0/1, pnpm to bun.lock 0/0, npm to bun.lockb 1/1, yarn to bun.lockb 1/1, npm to bun.lock 1/0, bun install --yarn on bun.lock 0/1.
  • save_to_disk on bun.lockb during bun add at 1,002 packages: 871,346 instructions, 18 allocations (base 871,273, 18). At 202 packages: 141,008 and 18 (base 141,010 to 141,700 and 18).
  • no-op bun.lock install at 202 packages: 5,931,191 instructions inside install_with_manager (base 5,939,051). Fresh --lockfile-only install to bun.lock: 5,389,872 (base 5,879,068).
  • binding.gyp probes per install: 1 (base 1). Per bun pm migrate to bun.lockb: 1 (base 0). Per bun pm migrate to bun.lock: 0 (base 0).
  • bun pm migrate to bun.lockb: clean_with_logger calls 1 (base 0). Save step at 202 packages: pnpm 1,600,203 instructions and 52 allocations, npm 1,845,084 and 489 (base 146,858 and 18, 146,488 and 18). bun pm migrate to bun.lock: 1,893,448 instructions (base 1,893,390), same allocations.
  • known-answer files: 8 of 8 bun.lockb files written by the base build pass bun ci on the PR build. Bytes differing after a re-save by the PR build: 0. bun pm trust keeps the stored hash.
  • first-frozen sweep over every foreign-lockfile fixture in test/cli/install/migration, 275 rows: unchanged flips 0 of 97, dependency-deleted flips 0 of 81, root-hook rows 1 to 0: 52 (48 package-lock.json, 4 yarn.lock), 0 rows go 0 to 1.
  • bun.lockb producer rows failing the next frozen install: 0 of 123 (base 37 of 123). The matrix is pnpm, yarn, npm and no lockfile, times pm migrate, install, install --lockfile-only, add and remove, times seven project shapes (plain, root postinstall, root binding.gyp, root prepare plus binding.gyp, an unreferenced lock entry, a workspace postinstall, a workspace binding.gyp).
  • bun pm migrate to bun.lockb against bun install --lockfile-only on the same input, 14 cases: 14 store the same meta hash (base 3), 4 files are byte-identical (base 0). The others differ by one byte (configVersion, which #41908 changes) or two (a pnpm importer literal, which #38791 changes).
  • size_of::(): 1,304 bytes (base 1,448). Release binary: text 80,663,855 bytes (base 80,679,983), data and bss unchanged (size).
  • hash-string blocks printed by --verbose per flow (base/PR): fresh bun.lock 1/1, bun.lock no-op 0/0, fresh text --lockfile-only 1/1, bun.lockb no-op 1/1, bun.lockb frozen 1/1, bun.lockb --lockfile-only 1/1, fresh bun.lockb 1/1, pnpm to bun.lockb 0/0, npm to bun.lock 1/1. The blocks are identical.

Behaviour that changes.

  • A bun.lockb with all-zero hash bytes and more than one package loads as "no stored hash". A frozen install compares it with Lockfile::eql, the check bun.lock uses, and a plain install writes the hash. Before, the frozen install failed until a plain install ran. Files from the pnpm migration of 1.2.23 to 1.4.2 are in this state.
  • The first frozen install from package-lock.json or yarn.lock passes when package.json has lifecycle scripts. The hash of the migrated lockfile is now taken from the same columns as the hash after the resolve. A dependency that package.json dropped still fails the check.
  • bun install --yarn prints the hash of the lockfile in the yarn.lock header when it loaded a bun.lock. Before, it printed zeros. bun bun.lockb does the same for a file without a stored hash.
  • bun pm hash and bun pm hash-string are unchanged. They hash packages only.
  • bun pm migrate and bun pm trust no longer abort a debug build when a migrated pnpm-lock.yaml has patchedDependencies and the output is bun.lockb. Two debug assertions required a patch file hash that only an install computes. They are gone. #41908 removes the same one in the writer.

Differences from the design review.

  • The review wanted the direct save to compute patch file hashes. This PR leaves them unset, as a release build of main does. Every install recomputes them before it uses them.
  • The review expected the first frozen install from package-lock.json to keep failing when a workspace has install scripts. It passes here, because the loaded side of the comparison is hashed from the lockfile after the differ has read the workspace scripts. A test covers it.
  • The review expected 2/1 hash computations for an npm to bun.lock install. The count stays 2/2: one for the migrated lockfile, one for the result.

Not covered. bun pm migrate to bun.lockb from a yarn v1 workspace project with a workspace lifecycle script. The yarn migrator creates no workspace rows, so the direct save has nothing to fill. This is from reading the code. I did not measure it.

Overlap with open PRs. #43912 adds Scripts::wants_default_node_gyp. This branch adds the same helper with the same text, so the two merge with one trivial conflict in load_root_lifecycle_scripts. #44294 conflicts in one line of package_manager_command.rs. #41908 touches save_to_disk, bun pm migrate and bun pm trust. #40057 adds a migrator that assigns meta_hash. That line goes away with the field.

Suites run with the debug build. bun-lockb, bun-pm, bun-lock, bun-workspaces, bun-install, bun-install-registry, bun-install-lifecycle-scripts, frozen-lockfile-pruned, frozen-lockfile-missing-workspace, migrate-bun-lockb-v2, lockfile-version-2, lockfile-only, overrides, config-version, bun-update-lockfile-sync and all of test/cli/install/migration/. Failures that also occur without this change: the bitbucket, gitlab and external tarball tests in bun-install.test.ts need the network. ensureTempNodeGypScript works and node -p should work in postinstall scripts clear PATH and need a binary named bun. manifest conditional requests > a changed etag returns 200 fails about one run in three on the debug build. Several files need more than the 5 s default per test on a debug build.