test(install): speed up bun-install-lifecycle-scripts.test.ts and check what the scripts do
Closed
RO
1mo ago+207+207+012345678901234567890123456789-210-210-012345678901234567890123456789
Problem
bun-install-lifecycle-scripts.test.tstakes 40s on x64-asan, 7s on the default lane. ASAN builds run 5 tests at a time, so total work sets the wall time.- "stress test" starts the ASAN
bunbinary 500 times and "root package with all lifecycle scripts" 18 times: 35% of the work. Linux runs both twice. - Neither test checked that a script ran. Six other tests had weak or wrong assertions (list in Notes).
Fix
- The scripts are shell builtins now:
echo postinstall > postinstall.txt,echo preinstall >> preinstall.txt. The tests check all 500 marker files, and one line per run per root script after each install. - "reach max concurrent scripts" records how many scripts ran at once. Each must see at most one other. With the limit raised to 4, every script sees 3.
- The three PATH tests set the scripts' PATH themselves (empty, or one empty directory), send
bun xto the local registry and check what ran: thebuild.nodeof the registry's node-gyp, or theprocess.versions.bunof thenode.bunEnvintest/harness.tsnow dropsNODEandnpm_*, whichbun runexports (same hunk as #37384). Before, two of these tests failed underbun bd test. - Verified: on the x64-asan lane of this PR's CI run the file takes 22.2s (40s before).
bun bd test, alternated on one machine: Linux debug+ASAN 39.7s and 39.4s before, 25.1s and 24.4s after, CPU time 4m56s to 1m31s. Windows debug 22.5s to 18.3s.
Background
- Bun prints
$ <script>to stderr before each root script. New assertions read these lines for the order of the root scripts. - Scripts run through
sh -con POSIX and through bun's shell on Windows.echoand>>work the same in both. - The
node-gypshim runsbun x node-gyp, which installs outside the project, so onlyBUN_CONFIG_REGISTRYreaches it. bun installuses the node named byNODEbefore it searches PATH (get_node_path,src/dotenv/env_loader.rs). It then adds nonodeorbunof its own.
Notes
Assertion changes, per test:
- "root package with all lifecycle scripts": exact contents of the six root files after each install (one line per run, so a script that runs twice in one install, or not at all, fails), the exact ordered list of the six
$lines on all three installs, inline snapshots of stdout, and the dependency's six files as one object. Theall-lifecycle-scriptsdependency still runs itsbunscripts, so that path stays covered. - "stress test": all 500
postinstall.txtfiles with exact contents. The comment in the test already said the script only has to be quick. - "reach max concurrent scripts": every package's
preinstall.txtexists and holds0or1. Each script writes a marker, sleeps 250ms, counts the other markers, sleeps 250ms more and removes its marker. Scripts that start together have all written their markers by the time any of them counts. With--concurrent-scripts=4instead of 2 every script recorded3in 6 of 6 runs. The script usesBun.write,Bun.GlobandBun.file().delete()becauserequire("fs")adds about 0.5s per script on a debug build. The total sleep is still 500ms, so the test costs the same as before. - "node -p should work in postinstall scripts": exact stderr lines, stdout snapshot (the
undefinedthatnode -pprints), and the script writesprocess.versions.bun, which only bun's ownnodehas, so the file has to hold the version of the binary under test. Both variants now run with an empty PATH. Before, the test setenv.PATH = ""aftertestEnvhad been copied fromenv, so the waiter thread variant ran with a full PATH. - "ensureTempNodeGypScript works" (empty PATH) and "node-gyp should always be available for lifecycle scripts" (a PATH with one empty directory on it, so
bun installtakes the branch that appends to an existing PATH): exact stderr lines, andbuild.nodehas to hold the package directory, which is what the registry's node-gyp writes. The first one also spawns withtestEnvnow (it usedenv, so the waiter thread flag was never set). Both used to download the real node-gyp from registry.npmjs.org throughbun x, and the second one used to search the host PATH, so a node-gyp installed on the machine (test/package.json has one) or thebunon the PATH took the place of the onesbun installprovides. With a stub node-gyp first on the PATH, the new tests pass and the first version of this PR failed. Both ignore stdout: on Windows the shim is a.cmdfile and cmd.exe echoes its lines to stdout (seen on the Windows run). envWithPathremoves every spelling of PATH before it setsPATH. On Windows the env carriesPath, a spawned bun reads both, and the last one it reads wins, so{ ...env, PATH: "" }does not reliably empty the PATH there. On Windows all three tests pass with the PATH really empty, so thenode.exeandbun.exelinks thatbun installcreates there are what these tests run.- "automatic node-gyp scripts work in package root": both installs check for the
$ node-gyp rebuildline, and the second install checks stderr and stdout instead of only the exit code. - "default trusted dependencies should work": spawns with
testEnvinstead ofenv.
Timing details:
- The per-test numbers come from a temporary wrapper around
setupTest()that recorded how long each test held a slot. Before: 233s of slot time in total, 5 slots busy for the whole run (the ASAN default fromsrc/options_types/context.rs, so the file's own 12-slot semaphore never binds on that lane), "root package" 30s + 21s, "stress test" 19s + 12s. After: 95s in total. - The slowest tests now are the two "stdout/stderr is inherited" cases (about 4s each on a debug build, three
bun -escripts each) and "auto node-gyp script does not drop the package's own postinstall" (threerequire("fs")scripts). Both are much cheaper on release builds, so they are left alone. - Full numbers. Linux debug+ASAN, 16 vCPUs on a shared host: before 39.7s and 39.4s (119 pass, 3 fail), after 25.1s and 24.4s (122 pass). CPU time (user+sys) 2m05s+2m51s before, 1m03s+0m28s after. An earlier pair under more load measured 54.6s before and 25.4s after. Windows x64 debug: before 22.5s and 22.8s (2 fail), after 18.3s and 19.8s (64 pass, 1 todo). On Windows the stress test still takes about 14s because every script is a
bun execprocess there. This PR does not change that. - CI, build 101254 of this PR, time of this file per lane: x64-asan 22.2s (39.9s in
test/expected-durations.json), windows 2019 x64 11.8s (15.1s), windows 11 aarch64 14.0s (14s), alpine x64 8.3s (8.7s), debian x64 9.9s (7.5s), debian and ubuntu aarch64 8.6s and 9.7s. Build 101434 (second commit): x64-asan 21.7s, windows 2019 x64 12.7s, windows 11 aarch64 15.2s, alpine x64 5.9s, debian x64 8.3s. The runner puts a changed file at the front of its shard, while the shard's services still start, so these numbers are higher than the same file measures on main. On the release lanes the work that is left is mostly the installs themselves and the sleeps the tests depend on. - The failures of the old file under
bun bd testwere "node -p should work in postinstall scripts" and "ensureTempNodeGypScript works" on both platforms:bun run(so alsobun bd) exportsNODEandnpm_node_execpath, sobun installtook theget_node_pathbranch and the scripts found neithernodenorbun. - The second commit is the result of a self-review of the first one. It moved the
NODEstrip from a helper in this file intotest/harness.ts, where the otherbunEnvstrips are, and made the three PATH tests independent of the host PATH as described above. - The harness hunk applies to every test file, so for the record: CI starts
bun testdirectly and never has these variables, so it changes nothing in CI. Locally it makesbun bd testmatch CI. The hunk is byte for byte the one in #37384, so the two merge in either order. It also makesshould set "npm_config_user_agent" to buninbunx.test.tspass underbun bd test(it saw the outer bun's value before).test/regression/issue/26207.test.tsclears the same two variables for itself. That stays as it is, it does no harm. - #37384 edits the same two PATH tests for another reason (the debug-only wipe of the shim dir). The conflict is limited to those lines.
- After the second commit: Linux debug+ASAN 122 pass (28.3s, the machine was busier), Windows x64 debug three full runs of 17.9s to 18.0s, 64 pass and 1 todo each. One earlier Windows run hung in the first
bun installof "add trusted, delete, then add again > withRm" until the 5 minute timeout, a test this PR does not touch, with nothing written to its directory yet. Five runs of that test alone and the three full runs after it were clean. - Not changed:
setDefaultTimeout, the waiter thread loop, the 1s sleeps in the fixtures that the "wait for dependency scripts" and "does not use 100% cpu" tests measure against, the git dependency test (#39115 makes it local), and thetestEnvline that every test in the block repeats (foldingforceWaiterThreadintosetupTest()touches all 55 tests, so it is a change of its own).tscovertest/reports nothing for these files, before and after.