#44465 bun test: report the run before process.exit() ends it, and exit 1 when a test failed

The rule

It applies where test files run in the main process: the default runner, --parallel=1, and --parallel with one file.

The report is the code that the normal end of a run executes: inline and .snap snapshot writes, the coverage table, coverageThreshold and lcov, the summary, the JUnit file, the timings file. The two --bail exits are not touched.

The lists: this file did not finish registering its tests (an exit at the top level or in a describe callback), N tests in this file did not finish, N hooks in this file did not run, and the files that did not start (10 names, then a count). A test with no callback (skip, todo, filtered out by -t) does not count.

Example, bun test ./a.test.ts ./b.test.ts, no failed test:

a.test.ts:
note: process.exit() was called by the test "exits" while a.test.ts was running (exit code 0)
1 test in this file did not finish
1 test file did not run:
  b.test.ts
note: bun test exits when the run ends. To test code that calls process.exit(), mock it with spyOn(process, "exit")

 0 pass
 0 fail
Ran 0 tests across 1 file.

After a failed test:

(fail) fails
error: process.exit() was called by the test "exits" while a.test.ts was running (exit code 0). The test run has failed, so bun test exits with code 1.
      at <anonymous> (/tmp/x/a.test.ts:3:31)

A test that failed in its body and still runs its afterEach hook gets its (fail) line and its JUnit case before the report. Without that, its failure is not counted yet when the hook exits.

Question for a maintainer

Should an exit that cuts the run short fail the run, also when no test failed? This PR says no: exit code 0 stays. That is what serial bun test does today, and jest -i and node --test keep the code of a deliberate exit too. bun test --parallel and vitest exit 1. For files that import from vitest (draft #44833), a third option is what vitest does: process.exit() throws inside a test.

The strict rule is a small change on top of this PR: count every exit that reaches report_run_ended_by_exit as one failed test, as it already does for a non-zero code. It would turn these green runs red:

• afterAll(() => process.exit(0)) in a file that is not the last file.
• process.exit(0) as the last statement of a test.
• A file that skips itself with a top-level process.exit(0).
• A SIGTERM listener that exits 0: a killed serial run exits 0 today, and with this PR it still does, with a report.

An earlier head of this PR had a rule in between (fail only when the exit leaves work undone). The self-review found that its predicate was wrong in both directions, and that the exit code of a suite then depended on the order of its files. It is gone.

The other open PR

#44503 (same account, opened nine hours after this one) fixes the first half of this bug with the same exit-code rule. This PR took three things from it: the vm.is_main_thread check, the report of a failed test that still runs its hooks, and the --bail handling in the exit report. One thing is only in #44503: a hook in Process_functionExit that tells 'exit' listeners code 1 before they run. The two PRs conflict in six files, so one of them must be closed. I did not close #44503.

Where this came from

An automated fuzz run found it. No user issue reports it by this cause. #30370 (closed with no reply) reports bun test printing only the version line and exiting 0. The code under test there calls process.exit(0) (TuiApp.ts:110 at 8ef0a17, from a key handler through a timer). I read that source and did not run it: it needs Windows and a native build.

Self-review

Nine concerns survived the review. Eight are addressed, one in part:

1. The exit-code rule had four forms in two days and was wrong in both directions. One structural rule replaces it: only a non-zero code makes the exit a failure.
2. The hook also ran for VMs that do not own the run. bun test --inspect-wait on a taken port printed a false failure from the Debugger VM, and a macro inside Bun.build() was reported too. on_requested_exit now checks vm.is_main_thread (the field, as #44503 did). Tests cover both.
3. The count of unfinished tests included tests that can never run and ignored hooks. Execution::unfinished counts from the stepper's cursor.
4. A non-zero exit could end a run whose report said that everything passed. It now counts as one failed test.
5. The exemption for a lone file that registered nothing is removed. The vendored node tests set BUN_TEST_DRAIN_EVENT_LOOP=1 and never reached it, and it kept the original hole open for bun test one-file.ts.
6. In part: the message showed the code as an argument the user did not write (after % 256, after 'exit' listeners). It now says process.exit() was called ... (exit code N) and names the test or hook when the call is in tail position and leaves no frame. Not done: telling 'exit' listeners the final code. They run before Bun__Process__exit, and the final code depends on the report (coverage threshold, snapshot writes), which runs after them.
7. The error gave no reason and no remedy. The report now has a line for a file that did not finish registering its tests, and ends with a note: that names the remedy.
8. Tests: whole-output snapshots, JUnit totals, inline snapshots, lcov, --dots, --only-failures, concurrent tests, a killed run. Not added: --randomize, --bail=2.
9. The list of files that did not run is capped at 10 names.

The review also found that a failed test which still runs its hooks was missing from the (fail) lines and the JUnit file, and that BUN_TEST_DRAIN_EVENT_LOOP=1 exited 0 for it. Execution::end_failed_sequences_in_flight reports it first (approach from #44503).

After that rework, a review bot and the session that works on #44503 found one more case: when such a test reached the --bail count, handle_test_completed ended the process inside the exit report, before vm.on_exit(). A buffered file write was lost and a non-zero code became 1. The exit report now applies --bail itself and prints the Bailed out line (approach from #44503).

Tests

• test/cli/test/bun-test.test.ts, the process.exit() in a test file block: 58 tests. On 1.4.3-canary.1, 51 fail. The 7 that pass there pin behaviour that must not change (BUN_TEST_DRAIN_EVENT_LOOP=1 with the code of the exit, a Worker, the Debugger VM, a macro VM, a mocked process.exit).
• The block runs serially on ASAN builds. Each test starts bun test, and 20 ASAN processes at once timed out on a 15 GB machine.
• The whole file on a debug build of this branch: 160 pass, 6 todo, 2 fail. The 2 failures are the nested too deeply to print are a RangeError tests from #44353. They time out at 5 s on the debug build, also on main alone.
• test/cli/test/parallel.test.ts on the same build: 45 pass, 1 skip. In 1 of 3 runs each worker has a unique JEST_WORKER_ID and BUN_TEST_WORKER_ID failed: two files ran on one worker. It passes alone, and this PR does not touch the --parallel code.

Measurements

Release builds, Linux x64: main f0c6cbc81 against this branch at 5fb78248d, which is that main plus this PR. Both come from one build directory. Only the five changed source files differ.

• Size (size -A, nm -S): .text 81,171,205 to 81,178,373 bytes (+7,168). The other loaded sections are equal. The stripped binary grows by 8,192 bytes. Bun__Process__exit 212 to 242 bytes. TestCommand::exec 22,098 to 16,212 bytes. New: report_run 9,051, report_run_ended_by_exit 5,188, exit_after_report 608, end_failed_sequences_in_flight 299, on_requested_exit 93 bytes. global_exit 89 to 94, run_all_tests 189 to 228 bytes.
• process.exit() outside bun test (objdump): +16 instructions (6 in Bun__Process__exit, 6 in on_requested_exit, 4 in global_exit), +3 loads, +2 branches, 0 syscalls, 0 allocations. perf and valgrind are not in the build container, so there is no perf stat line.
• A run that does not exit early (objdump): run_all_tests gains 4 stores per run. Execution::advance_sequence, which runs once per test, has 11,753 bytes in both builds. TestCommand::run, which runs once per file, has 7,897 bytes in both builds.
• Data: ExitHandler gains 1 byte. TestRunner gains 24 bytes, once per process.

Not in this PR

• Files after the exit do not run. That needs a process boundary.
• The two --bail exits still write a subset of the report (#38974, #38265, #39010). After this PR they can call report_run.
• An 'exit' listener that throws at the normal end of a run is printed and not counted (#41588).
• A --parallel worker that calls process.exit() never reaches worker_flush_aggregates, so the coverage, snapshots and failure recap of the files it ran before are lost (read in src/runtime/cli/test/parallel/runner.rs, not run).
• A non-zero exit between two files under --isolate adds a second JUnit suite with the name of the file before it.
• A per-VM slot on ExitHandler in place of the is_main_thread check and the global runner. That is a new extension point, so it needs a maintainer's yes first.
• #40235 and the draft #44833 change the same functions of test_command.rs. Neither changes Bun__Process__exit.