#44503 bun test: report a run that has already failed, and do not exit 0, when process.exit() ends it

State

This PR is a draft. It waits for the answers to the questions below. #46639 has the part that needs no decision: a run that --bail ends keeps exit code 1. This branch has the same flag (ExitHandler.exits_nonzero) and the same store in Bun__Process__exit. When #46639 lands first, a merge of main into this branch has two conflicts: the comment of the flag, and bail_out, which this PR splits.

Where this came from

An automated fuzz run found it. No issue on this repository names the cause. Users of bun test reported it in their own projects (not linked here, to keep this PR out of their timelines):

• garrytan/gstack#2421: "exits 0 while tests are failing, having run roughly 23 of 409 test files. It never prints a summary line."
• garrytan/gstack#2435: "EXIT=0 with 82 (fail) lines and no tally block". The cause there is setTimeout(() => process.exit(0), 500) in a test file.
• garrytan/gstack pull request 2252, chudeemeke/get-stuff-done#45 and amadeus-dlc/amadeus#3280: a process.exit(0) in one test file ends the run for every file.

The first two have failures before the exit. This PR makes such a run exit 1 with its report. The others describe the run that ends early with no failure recorded. This PR does not change that (question 1).

The rule

It applies while TestCommand::run_all_tests runs the files in the main process: plain bun test, --parallel=1, and --parallel with one file.

The run has failed when a test or a hook has failed and has no retry left, or when an error was thrown between tests. --bail does not apply once an exit ends the run: every test that has failed is reported, and a non-zero code stays.

$ bun test
a.test.js:
(fail) a FAILS [0.12ms]

b.test.js:
usage: ...
1 | export function main(argv) { if (argv.includes("--help")) { console.log("usage: ..."); process.exit(0); } }
                                                                                                   ^
error: The process was told to exit with code 0 while the test "--help prints the usage" was running. The test run has failed, so the exit code is 1.
      at main (/tmp/cli/main.js:1:96)
      at <anonymous> (/tmp/cli/b.test.js:3:41)
note: bun test exits when the run ends, so a test file does not need to end the process
note: To test code that calls process.exit(), mock process.exit

 0 pass
 1 fail
 1 expect() calls
Ran 1 test across 2 files. [15.00ms]
$ echo $?
1

The message says what was on the stack, not who called. A timer of an earlier test can fire while a later test waits in .resolves.

Other runners

Exit codes, measured in one container. F: a test has failed before the exit. N: no test has failed. 0 and 3: the code given to process.exit().

Questions for a maintainer

1. The N0 column. A process.exit(0) before any failure still ends the run with exit code 0, and the same suite exits 1 with --parallel=2. The options:
   • Keep it (this PR). No run that is green today becomes red.
   • Fail every exit that ends the run, as --parallel, jest with workers and vitest do. It is a change of the predicate in on_process_exit. Green today and red then: afterAll(() => process.exit(0)), process.exit(0) as the last statement of a test, a file that skips itself with a top-level process.exit(0), a SIGTERM listener that exits 0.
   • Throw from process.exit() inside a test file, as vitest does. The run continues. It is a separate change in Process_functionExit.
   • Decide by the module that the test imports from, as #44833 does for other rules.
2. The order with #44833. A merge of this branch with its head 2705a1541f has one conflict more than a merge of main 6292c6539d with it: the count lines that this PR moves into print_counts and that #44833 rewrites in place. After that, TestRunner::on_stack must follow the rename of on_stack_entry_data, and has_failed must follow flip_fails. #46639 merges with no conflict.
3. The normal end. An 'exit' listener of a node:test file that stores 0 at the normal end of a failed run makes it exit 0 (1.4.2 and main, 1.3.13 exits 1). #46639 closes this for --bail only.
4. Snapshot files and coverage are not written when an exit ends the run, as for --bail today. If --bail starts to write them (#38974, #38265), does an exit follow?

Still exits 0 after this PR

• An exit before any failure is recorded (question 1).
• A failure that the runner records only when the test ends: an unmet expect.assertions() count, a test.failing test that passes, a subtest of a node:test test. An afterEach that calls process.exit(0) after such a test exits 0. So does a node:test afterEach after a failed test body.
• A macro of Bun.build() that calls process.exit(0). It runs in the VM of the bundler thread, which does not own the run.
• process.abort(), process.kill(process.pid), process.execve() and signals do not pass Bun__Process__exit.

The 'exit' listeners and process.exitCode read the 0 of the caller, and the process exits 1.

What changed in this PR since its first form

The first form of this PR had a hook in Process_functionExit with an ExitCodeState in Process, and changed only exits with code 0. A review of this PR and of #44465 led to the present form:

• Kept from the first form: the rule (only a run that has already failed changes), has_observed_failure, end_failed_sequences_in_flight, and most tests. Taken from #44465: the gate on the in-process run and the cases that record what does not change.
• Dropped: all C++, the --bail workaround (the bail_out of #46021 runs the exit steps itself), the snapshot tally, and the exit-code floor of #44465. The hook in Process_functionExit told the 'exit' listeners 1 before it was known that the exit was real (a process.reallyExit stub returns), and it added work to every process.exit().
• Added: the report for a non-zero code, the retry rule, the repeat of the failed tests of a long run, and neutral wording.
• The mark of a run that ended early is ExitHandler.exits_nonzero, as in #46639. It was a field of TestRunner. An 'exit' listener that exits then read the runner while end_early held the reporter.

The self-review raised 12 concerns on the code. Addressed: wording by what is on the stack, quoted names, notes that fit every exit, node:test cases, the repeat of failed tests, the --parallel paragraph in the docs, no manual walk of the entries, cases for the clauses that had none, --watch cases, the SAFETY comment, the snapshot tally, the doc comment. In part: the message names the test, not its describe path or the kind of hook. An unwind was also weighed (Bun__Process__exit stops the VM and returns, the tail of exec reports): same output, stop checks at about 10 runner sites, and a missed one is a hang. It is not built.

Measurements

Release builds, Linux x64, main 5c6d4e24cd against that main plus the src/ of this PR at 2854f713ec, from one build directory. The src/ diff is the same on main 6292c6539d.

• process.exit() outside bun test: Bun__Process__exit 13 to 20 instructions before call on_exit (+2 loads, +3 branches), 21 for code 0 (one more load). 212 to 254 bytes. +0 calls, +0 allocations, +0 syscalls.
• 0 changed lines (objdump, addresses and field offsets masked): global_exit 23 instructions, on_exit 143, Process_functionExit 200, Process_functionReallyExit 100, advance_sequence 2,329, run_test_callback 481, step_sequence 285, TestCommand::run 1,588, Expect::snapshot 1,712.
• Per run: run_all_tests 56 to 61 instructions.
• Release .text: 82,231,705 to 82,233,241 bytes (+1,536). The stripped binary has 107,808,480 bytes in both builds. TestCommand::exec 21,703 to 18,427. New: print_counts 2,200 and print_repeats 765 (both moved out of exec), end_by_process_exit 898, end_early 257, on_process_exit 190, end_failed_sequences_in_flight 187, the OnStack formatter 155. bail_out 311 to 222.
• TestRunner has 592 bytes and CommandLineReporter 968 in debug builds of both: the new bool field fills padding. ExitHandler 3 to 4 bytes. VirtualMachine has 33,024 bytes in both. Native exports +0.
• The exit that ends a failed run (main thread, from Bun__Process__exit to the end of the process): +25 mimalloc calls and +7 syscalls. With --reporter=junit: +24 and +10. Counted with gdb, 2 runs each.
• New cases: 35 of 47 fail on main and 47 of 47 pass here, on a debug ASAN build and on a release build. BUN_JSC_validateExceptionChecks=1 reports nothing on them.
• 24 fixtures that must not change give the same stdout, stderr (times masked) and exit code on both release builds. They include a long run that prints the two moved blocks. 26 more fixtures end a failed run by an exit: each prints the new report.
• strace, perf, valgrind and bloaty are not installed in the build container. The instruction counts are from objdump.

Tests

• test/cli/test/bun-test.test.ts, the block an exit of the process in a run that has failed. The whole file on a debug build: 155 pass, 6 todo, 2 fail. The 2 failures are the nested too deeply to print are a RangeError cases. They time out at 5 s on a debug build, also with the src/ of main.
• test/cli/test/parallel.test.ts: 47 pass, 1 fail. each worker has a unique JEST_WORKER_ID and BUN_TEST_WORKER_ID fails in this container, also with the src/ of main.
• isolation.test.ts 50 pass, rerun-each.test.ts 6, retry-flag.test.ts 9, test-shard.test.ts 29, coverage.test.ts 43, pass-with-no-tests.test.ts 5, test-timeout-behavior.test.ts 2.