#42001 install: list unfinished lifecycle scripts per project, so the install after a killed one runs them

What main does today (canary ff760aec5, Linux x64, loopback registry, one trusted package whose postinstall blocks, both linkers):

The released 1.4.3 behaves the same for the signals. bun install has no SIGINT handler: on Ctrl-C bun and the script both die of the signal.

The file. Records are lines. <path> is the package directory from the project's root: node_modules/held (hoisted), node_modules/.bun/held@1.0.0/node_modules/held (isolated). With the hoisted linker, a package in the tree of a workspace has the path of its folder (packages/b/node_modules/dep), not a path through the link to the workspace: a path through the link goes with the link. The file is opened with O_APPEND | O_NOFOLLOW | O_CLOEXEC, so a script does not inherit the lock. A path from the file is only compared with the paths of the packages of the pass, or given to fstatat. It is never a delete or rename target. A record that is empty, absolute, or has a .. or a NUL is not read.

• An empty file names nothing. A line that a dead install did not end is cut off when the file is opened, and is not acted on: a line is one write, made before its package is placed, so after a kill it names nothing that is in place.
• A file that cannot be read is left as it is, like a file that another install holds. So is a file of more than 1 MiB, which no install writes.
• A write that fails prints warn: failed to record pending lifecycle scripts in node_modules/.bun-pending-scripts: <error> once and stops the writing for the pass.
• A package whose scripts never run gets no line: one that the postinstall optimizer leaves out (esbuild and sharp by default), a member outside the project, a folder inside its declarer.
• A package that was in place when its name became trusted gets no line: the lockfile is saved after the pass, so the next install finds it newly trusted again and runs its script in place.

Where the code is.

• pending_scripts.rs: the list. In memory it is what a failed script discards (it replaces the Rc<RefCell<Vec>> of the hoisted linker). The file is its copy for the next install while this install holds the lock.
• Both drivers (hoisted_install.rs, isolated_install.rs) load it after they know whether node_modules is new, and finish it at their end. A new node_modules has no file to look for. This also covers the security scanner's partial install. A change of linker moves the file aside with the tree of the other linker.
• Replay: one pass before the walk, only when a dead install left records. It calls remove_placed_package (hoisted) or delete_store_copy (isolated) for each listed package of this pass that still runs scripts. Both skip checks are as on main. With the hoisted linker a nested tree is behind the link of a workspace in its parent tree, so the pass goes through nested trees first.
• With the hoisted linker two trees can be one folder (a workspace that is a dependency under two names). Both rows have the same record, because the record has the path of the folder. It ends with the last of them.
• A failed script removes the listed packages nested trees first as well. On main the link of the workspace goes first, and the package behind it stays in place and is never built.
• Whoever takes the lock reads the file, also when the first record of a pass opens it: another install can have left it since the pass began.
• Begin: on the main thread before the package is placed, for trusted dependencies (hoisted) and for trusted entries of the isolated store. With the hoisted linker a workspace is listed too: before its link is placed if the lockfile has its scripts, and else when they are read, right after the link is placed (bun.lock has no scripts, and most workspaces have none). The trust lookup that finish_install and prepare_scripts made moves there. It runs once per linked package, and never for a package that is skipped.
• Close: where main forgets (forget_unfinished_scripts), at the returns for a package without scripts, in on_task_complete, and after a discard. When a script fails with the isolated linker, the record of an entry that finished after the last run_tasks ends too.
• A package whose link fails, or whose scripts cannot be listed, stays listed. The next install tries it again.
• A listed package that is not trusted any more keeps its files and leaves the list. With the global store it moves there, and leaves the list too. A listed package that the pass does not come to (--filter) stays listed while its directory exists.
• bun patch <pkg> puts a copy to edit in place of the package. prepare_patch removes that package's record, so a plain install does not replace the copy. It also removes the record of a package that the copy is nested in (hoisted: node_modules/q/node_modules/k), because that package goes with all that is in it. bun patch --commit installs the patched package and runs its scripts. The install that bun patch <pkg> starts with lists what it places, and does not act on what a dead install left: it is not to fail on a script of the package it prepares. --ignore-scripts does not read or write the file.
• load_root_lifecycle_scripts asks the root's directory listing, which is read already, for binding.gyp before it calls access. This is the system call that pays for the probe of the file.

Measurements (release builds of main 5866b5f13d and of this branch on it, one commit before the last, which changes one line for Windows; x64 Linux; system calls of the bun install process counted per name with a ptrace counter, script children left out; strace, perf and valgrind are not available in this container):

• Release binary: text 106,331,459 -> 106,358,851 (+27,392 bytes, size). PackageManager grows by 72 bytes (size_of::<PendingScripts>() is 64). The hoisted linker's per-pass Rc allocation is gone.
• Warm no-op install, both linkers: no file system call changes by name. The probe is one access, and the access of the root's binding.gyp is gone.
• Fresh install of one trusted package with a postinstall, both linkers: openat +1, flock +1, fstat +1, newfstatat +1, unlinkat +1, close +1, write +2 (hoisted 14 -> 16, isolated 17 -> 19), access -1.
• Fresh install of one trusted package whose scripts the postinstall optimizer leaves out (ignoreScripts, the branch of esbuild): access -1, nothing else. The same without a trusted package.
• The postinstall optimizer is asked once more for each trusted package that is placed (before the placement and where main asks).
• Taken for the revision before the merge of #45307, against its merge base 5f5799ed16, and not taken again: Lockfile::has_trusted_dependency calls (gdb breakpoint count on bun-profile): warm no-op 0 and 0; fresh install of 125 packages: hoisted 250 -> 125 without trustedDependencies, isolated 125 -> 125. Allocator calls on a warm no-op (mi_malloc): hoisted 415 -> 414, isolated 444 -> 444. More trusted packages: write 3 -> 5, 13, 23 with 1, 5, 10 packages (hoisted).
• Instructions: not measured. Both skip checks are untouched in the diff.
• The install after a SIGKILL: 1 script start, package built (main: (no changes), 0 starts). A second install while the first holds the lock: +1 openat, +1 flock, +1 close, 0 script starts, (no changes).

Tests. the install after a bun install that died (hoisted|isolated): 42 for the hoisted linker, 40 for the isolated one (5 of them skipped: they are about hoisted trees). One more per linker in the install after a failed script.

They fail on main (62):

• SIGKILL, SIGTERM, SIGINT, SIGHUP while a postinstall runs; a file: folder; a link: folder and a workspace (hoisted); bun ci; a postinstall that had not started; bun's own exit after the patch of another package does not apply; a script that outlives the install; --ignore-scripts then a plain install; --filter of another workspace then a full install.
• bun patch of another package that is killed.
• A package in the nested tree of a workspace whose own script has not run: after a kill, after a kill and then a --production install that does not come to it, and after a failed script (this one with the hoisted linker only, it is the order of main). A package that left the project before a bun.lock named it: the workspace that had it is built, and the record goes when the folder goes.
• With the hoisted linker, a workspace under two names: a kill between the two script runs of a package in it.
• A cut list and then an install that dies too; a list that cannot be written (ulimit -f, the warning); a package whose scripts are left out is not in the list of an install that runs.
• A list written by hand (no kill, so these run on Windows): it names the package in a whole line, in a whole line before a cut line, or in a cut line; it is empty; it is 1 MiB long; a cut line after the line of a package that a --filter install does not come to (the file is read back after that install wrote to it); a list that appears while an install runs (hoisted); a package under a workspace that has two names, and one behind a workspace link that is gone (hoisted); lines for a member outside the project and a folder inside its declarer, which end without a removal (hoisted); bun patch of a package that the list names.
• bun patch of a package inside the one that was not built (isolated), and behind the link of a workspace that was not built (hoisted).

They pass with and without (17): a package that is not trusted any more (also with the global store, and a link: folder with the hoisted linker), an install that runs meanwhile, an install with the other linker, a list of more than 1 MiB, bun patch after a killed install, bun patch of a package inside the one that was not built (hoisted: without the change nothing replaces the copy) and behind a workspace link (isolated), the workspace with the isolated linker, and the failed script under a workspace with the isolated linker. The kill tests need a POSIX shell and FIFOs, like the tests next to them, so they skip on Windows.

The review follow-ups were checked against the builds before them: the two names of a workspace, the link that is gone, the package that left the project (hoisted), the global store, and the scripts that are left out fail there. The test of the two script runs fails when the - line is written after the first run.

Not tested: a file that cannot be read, the second look at the lock when the file is replaced between the open and the lock, and the record of an isolated entry that finished after the last run_tasks when a script fails (it depends on timing).

Also run with the debug build, on the merge with main 5866b5f13d: bun-install-lifecycle-scripts, isolated-install, hoist, bun-patch, bun-workspaces, bun-install-staging, bun-link, bun-prune, bun-install-link-path, bun-install-native-binlink, bun-install, bun-pm, bun-add, bun-update, bun-remove, bun-install-patch, bun-security-scanner-workspaces, bun-install-security-provider, config-precedence, bun-create. One merge of main before, also bun-install-registry. All pass, with three exceptions that main has too: does not use 100% cpu > install (a CPU budget of 0.75 s, which the debug build does not meet), two git tests of bun-install-registry that need github.com, and the 5 s default timeout in hoist and isolated-install (they pass with a longer one).

cargo check -p bun_install passes for x86_64-pc-windows-msvc, aarch64-apple-darwin and x86_64-unknown-freebsd. Nothing here ran on Windows or macOS.

Not covered.

• A trusted link: dependency with the isolated linker. Since #45307 the isolated linker runs its scripts when no dependent links to its folder yet, and the dependents make those links while the scripts run. So the sign that the scripts ran is there before they finish. One record for the entry does not follow those links: the dependents differ between installs (--filter, --production), and links from dependents outside the pass stay. An attempt with such a record is in the history of this branch and is reverted. The hoisted linker is covered: there each link is a row of a tree.
• A package that was in place when it became trusted runs its scripts in place, with no line. If an install that runs no scripts (--ignore-scripts, --lockfile-only) saves the trust after a kill, a later install does not run them, as on main. A line for it made the next install replace a folder that the dead install had not placed, for example the copy that bun patch prepares.
• A record whose folder no install comes to any more (its dependency left package.json before a bun.lock named it, for example after a killed bun add) stays while the folder stays.
• With the hoisted linker, a workspace folder that is moved after an install died takes its packages along. Their records have the old path and are dropped, as on main.
• With the hoisted linker, when the link of a workspace is missing at the start of an install, a listed package behind it is installed again one install later: the removal goes through the link.
• A list of more than 1 MiB is not read and not written. The file stays until node_modules goes.
• The lock belongs to the bun process. A script that outlives it (macOS and Windows have no PR_SET_PDEATHSIG, and a grandchild survives on Linux) can still run when the next install places the package again, as under --force today. A test pins this. The alternative is to let the script inherit the locked descriptor.
• An install that runs while another one holds the lock does not write the file for its own packages.
• A file system without file locks: the file is not used, as on main.
• No fsync, and no state for a list that lost bytes: after a power cut, or after the warning of a failed write, a placed package can be without a record. An earlier revision read an empty or cut list as "may have listed any package" and placed every trusted package again. That state could not be kept across a --filter install or bun patch without a mark of its own, and while it lasted an install replaced the copy that bun patch prepares.
• The record of a package is dropped when bun patch prepares a copy inside it (hoisted, nested node_modules): that package is then not built, as on main.
• A kill in the moment between the link of a workspace and its line (hoisted, scripts not in the lockfile) leaves the workspace without a record.
• Packages placed under --ignore-scripts are as on main: not listed, and a later plain install does not run their scripts.
• With the isolated linker, the .bin links of a dependent that is not trusted are not made again after its dependency's script failed. This is the same on main.
• bun pm trust <pkg> does not use the list. It records the trust only after the scripts finish, so a killed run can be repeated (checked on the canary).

Merge of main. The branch has main with #45307 and #45460. What changed here because of them: enqueue_lifecycle_scripts_for_trusted is main's again (a package that it returns early for is not listed in the first place), the tree path takes the hoisted rows, and .bun-pending-scripts is among the entries that a change of linker moves aside.

History. Earlier revisions of this PR removed the package when its script failed (now in main through #44968), then marked each package with an empty file. A pre-merge check of the mark found that a second install reinstalled a package under a running script, that bun patch was blocked, and that every no-op install paid a trust lookup per package. This revision replaces the mark. Two review rounds of it (the bot's and a self-review of the follow-up) found that the "lost bytes" state was the cause of most defects, that the link of a workspace went before the packages behind it, that I/O errors were not handled, and that the install of bun patch listed nothing. The follow-up removes the state and changes the rest. Four more self-review rounds on the merge with main (57 concerns, 11 of them on the link: attempt) found the two trees for one folder, the records of packages whose scripts never run, the global store, and that neither the link: entries of the isolated linker nor the packages that were in place hold with one record each. They also found that a record with a path through the link of a workspace goes with that link, and that the rule that kept such a link in place left a workspace unbuilt where main removes it. The record has the path of the folder now, and the rule is gone. The optional-dependency guard and the hoisted link: test of the earlier revisions are gone: main removes exactly what the linker placed.