test(install): speed up bun-install-lifecycle-scripts.test.ts and check what the scripts do

Closed
RO
1mo ago
+207-210

Problem

  • bun-install-lifecycle-scripts.test.ts takes 40s on x64-asan, 7s on the default lane. ASAN builds run 5 tests at a time, so total work sets the wall time.
  • "stress test" starts the ASAN bun binary 500 times and "root package with all lifecycle scripts" 18 times: 35% of the work. Linux runs both twice.
  • Neither test checked that a script ran. Six other tests had weak or wrong assertions (list in Notes).

Fix

  • The scripts are shell builtins now: echo postinstall > postinstall.txt, echo preinstall >> preinstall.txt. The tests check all 500 marker files, and one line per run per root script after each install.
  • "reach max concurrent scripts" records how many scripts ran at once. Each must see at most one other. With the limit raised to 4, every script sees 3.
  • The three PATH tests set the scripts' PATH themselves (empty, or one empty directory), send bun x to the local registry and check what ran: the build.node of the registry's node-gyp, or the process.versions.bun of the node. bunEnv in test/harness.ts now drops NODE and npm_*, which bun run exports (same hunk as #37384). Before, two of these tests failed under bun bd test.
  • Verified: on the x64-asan lane of this PR's CI run the file takes 22.2s (40s before). bun bd test, alternated on one machine: Linux debug+ASAN 39.7s and 39.4s before, 25.1s and 24.4s after, CPU time 4m56s to 1m31s. Windows debug 22.5s to 18.3s.

Background

  • Bun prints $ <script> to stderr before each root script. New assertions read these lines for the order of the root scripts.
  • Scripts run through sh -c on POSIX and through bun's shell on Windows. echo and >> work the same in both.
  • The node-gyp shim runs bun x node-gyp, which installs outside the project, so only BUN_CONFIG_REGISTRY reaches it.
  • bun install uses the node named by NODE before it searches PATH (get_node_path, src/dotenv/env_loader.rs). It then adds no node or bun of its own.
Notes

Assertion changes, per test:

  • "root package with all lifecycle scripts": exact contents of the six root files after each install (one line per run, so a script that runs twice in one install, or not at all, fails), the exact ordered list of the six $ lines on all three installs, inline snapshots of stdout, and the dependency's six files as one object. The all-lifecycle-scripts dependency still runs its bun scripts, so that path stays covered.
  • "stress test": all 500 postinstall.txt files with exact contents. The comment in the test already said the script only has to be quick.
  • "reach max concurrent scripts": every package's preinstall.txt exists and holds 0 or 1. Each script writes a marker, sleeps 250ms, counts the other markers, sleeps 250ms more and removes its marker. Scripts that start together have all written their markers by the time any of them counts. With --concurrent-scripts=4 instead of 2 every script recorded 3 in 6 of 6 runs. The script uses Bun.write, Bun.Glob and Bun.file().delete() because require("fs") adds about 0.5s per script on a debug build. The total sleep is still 500ms, so the test costs the same as before.
  • "node -p should work in postinstall scripts": exact stderr lines, stdout snapshot (the undefined that node -p prints), and the script writes process.versions.bun, which only bun's own node has, so the file has to hold the version of the binary under test. Both variants now run with an empty PATH. Before, the test set env.PATH = "" after testEnv had been copied from env, so the waiter thread variant ran with a full PATH.
  • "ensureTempNodeGypScript works" (empty PATH) and "node-gyp should always be available for lifecycle scripts" (a PATH with one empty directory on it, so bun install takes the branch that appends to an existing PATH): exact stderr lines, and build.node has to hold the package directory, which is what the registry's node-gyp writes. The first one also spawns with testEnv now (it used env, so the waiter thread flag was never set). Both used to download the real node-gyp from registry.npmjs.org through bun x, and the second one used to search the host PATH, so a node-gyp installed on the machine (test/package.json has one) or the bun on the PATH took the place of the ones bun install provides. With a stub node-gyp first on the PATH, the new tests pass and the first version of this PR failed. Both ignore stdout: on Windows the shim is a .cmd file and cmd.exe echoes its lines to stdout (seen on the Windows run).
  • envWithPath removes every spelling of PATH before it sets PATH. On Windows the env carries Path, a spawned bun reads both, and the last one it reads wins, so { ...env, PATH: "" } does not reliably empty the PATH there. On Windows all three tests pass with the PATH really empty, so the node.exe and bun.exe links that bun install creates there are what these tests run.
  • "automatic node-gyp scripts work in package root": both installs check for the $ node-gyp rebuild line, and the second install checks stderr and stdout instead of only the exit code.
  • "default trusted dependencies should work": spawns with testEnv instead of env.

Timing details:

  • The per-test numbers come from a temporary wrapper around setupTest() that recorded how long each test held a slot. Before: 233s of slot time in total, 5 slots busy for the whole run (the ASAN default from src/options_types/context.rs, so the file's own 12-slot semaphore never binds on that lane), "root package" 30s + 21s, "stress test" 19s + 12s. After: 95s in total.
  • The slowest tests now are the two "stdout/stderr is inherited" cases (about 4s each on a debug build, three bun -e scripts each) and "auto node-gyp script does not drop the package's own postinstall" (three require("fs") scripts). Both are much cheaper on release builds, so they are left alone.
  • Full numbers. Linux debug+ASAN, 16 vCPUs on a shared host: before 39.7s and 39.4s (119 pass, 3 fail), after 25.1s and 24.4s (122 pass). CPU time (user+sys) 2m05s+2m51s before, 1m03s+0m28s after. An earlier pair under more load measured 54.6s before and 25.4s after. Windows x64 debug: before 22.5s and 22.8s (2 fail), after 18.3s and 19.8s (64 pass, 1 todo). On Windows the stress test still takes about 14s because every script is a bun exec process there. This PR does not change that.
  • CI, build 101254 of this PR, time of this file per lane: x64-asan 22.2s (39.9s in test/expected-durations.json), windows 2019 x64 11.8s (15.1s), windows 11 aarch64 14.0s (14s), alpine x64 8.3s (8.7s), debian x64 9.9s (7.5s), debian and ubuntu aarch64 8.6s and 9.7s. Build 101434 (second commit): x64-asan 21.7s, windows 2019 x64 12.7s, windows 11 aarch64 15.2s, alpine x64 5.9s, debian x64 8.3s. The runner puts a changed file at the front of its shard, while the shard's services still start, so these numbers are higher than the same file measures on main. On the release lanes the work that is left is mostly the installs themselves and the sleeps the tests depend on.
  • The failures of the old file under bun bd test were "node -p should work in postinstall scripts" and "ensureTempNodeGypScript works" on both platforms: bun run (so also bun bd) exports NODE and npm_node_execpath, so bun install took the get_node_path branch and the scripts found neither node nor bun.
  • The second commit is the result of a self-review of the first one. It moved the NODE strip from a helper in this file into test/harness.ts, where the other bunEnv strips are, and made the three PATH tests independent of the host PATH as described above.
  • The harness hunk applies to every test file, so for the record: CI starts bun test directly and never has these variables, so it changes nothing in CI. Locally it makes bun bd test match CI. The hunk is byte for byte the one in #37384, so the two merge in either order. It also makes should set "npm_config_user_agent" to bun in bunx.test.ts pass under bun bd test (it saw the outer bun's value before). test/regression/issue/26207.test.ts clears the same two variables for itself. That stays as it is, it does no harm.
  • #37384 edits the same two PATH tests for another reason (the debug-only wipe of the shim dir). The conflict is limited to those lines.
  • After the second commit: Linux debug+ASAN 122 pass (28.3s, the machine was busier), Windows x64 debug three full runs of 17.9s to 18.0s, 64 pass and 1 todo each. One earlier Windows run hung in the first bun install of "add trusted, delete, then add again > withRm" until the 5 minute timeout, a test this PR does not touch, with nothing written to its directory yet. Five runs of that test alone and the three full runs after it were clean.
  • Not changed: setDefaultTimeout, the waiter thread loop, the 1s sleeps in the fixtures that the "wait for dependency scripts" and "does not use 100% cpu" tests measure against, the git dependency test (#39115 makes it local), and the testEnv line that every test in the block repeats (folding forceWaiterThread into setupTest() touches all 55 tests, so it is a change of its own). tsc over test/ reports nothing for these files, before and after.