bun test: write coverage reports when --bail stops the run

RO
2mo ago
+208-27

Problem

  • bun test --bail --coverage emits no coverage at all once bail trips: no text table, no lcov.info, the coverage directory is not even created. The JUnit report and --update-timings file for the same run are written.
  • Coverage is only generated on the normal end-of-run path (TestCommand::exec, src/runtime/cli/test_command.rs, the generate_code_coverage call before the pass/fail counts). The two serial bail exits skip that path entirely:
    • CommandLineReporter::handle_test_completed (a test failure reaches --bail): print_summary, "Bailed out", JUnit, timings, Global::exit(1).
    • TestCommand::run_one_file (a test file fails to load, e.g. a top-level throw or a syntax error): same sequence, then global_exit().
  • --parallel is not affected: Coordinator::bail_out only stops dispatching files, and the coordinator still merges the workers' coverage chunks after the run loop ends.
  • Long-standing (same on 1.3.x); jest --bail writes coverage for what ran.

Fix

  • Adds CommandLineReporter::bail_out(vm), which is what both serial bail sites now call: flush, coverage reporters, summary line, "Bailed out" message, JUnit, timings. Each caller still exits the way it did before (Global::exit(1) mid-file, VM teardown plus global_exit() on load failure), so the only behavioral change is the coverage output.
  • bail_out calls the existing generate_code_coverage(vm, opts) on a clone of the configured CodeCoverageOptions when coverage is enabled. The normal end-of-run path in exec still calls it the same way, so its fractions.failing exit-code check is unchanged.
  • generate_code_coverage now returns the lcov.info write error instead of exiting inside: exec keeps the old print-and-exit-1 behavior, while bail_out prints the error and still writes the summary, the "Bailed out" message, the JUnit report and timings (the process exits 1 right after either way).
  • generate_code_coverage / for_each_coverage_report now take &VirtualMachine: they only read vm.global(), and the mid-file bail site has no &mut VirtualMachine in scope (it uses VirtualMachine::get()).
  • The report is correct at bail time because coverage data is read from JSC's control-flow profiler and the thread-local ByteRangeMapping table for whatever files were loaded so far; nothing about it depends on the run having finished. The table/lcov produced on bail are byte-identical to what a non-bail run of the same executed code produces (checked on both this build and the release binary).
  • Verified:
    • test/cli/test/coverage.test.ts, new --bail block: one test per bail site, plus one where --coverage-dir points at a plain file so the lcov write fails and the JUnit report and bail message must still appear. The first two fail on the current release (table missing from stderr, cov/lcov.info absent) and pass with this change. The inline lcov snapshot matches what the release binary produces for the same executed code without --bail.
    • The spawned child runs with detect_leaks=0 (same pattern as other tests that spawn into a known leaky exit): the mid-file bail exits through Global::exit without VM teardown, so on the ASAN lane LSan appends a report to stderr and aborts. That is #32183 and happens without --coverage too (checked locally: bun test --bail alone reports the same leak set under detect_leaks=1); a passing --coverage run under BUN_DESTRUCT_VM_ON_EXIT=1 is LSan-clean, so the coverage code itself is not leaking.
    • bun bd test test/cli/test/coverage.test.ts test/regression/issue/26851.test.ts test/regression/issue/12250.test.ts (26851 covers JUnit-on-bail through the refactored sites).
    • bun bd test test/cli/test/bun-test.test.ts -t bail, test/js/bun/test/test-test.test.ts -t bail (the latter exercises the load-failure bail site), test/cli/test/parallel.test.ts -t "coverage|bail".
    • Serial coverageThreshold exit codes unchanged (text reporter exits 1, lcov-only still exits 0 as before; that gap is #32118).

Background

  • --bail[=N] makes bun test stop after N test failures. A test file that throws while being loaded counts as one failure.
  • --coverage makes the transpiler register every loaded (non-test, by default) file in a ByteRangeMapping table and turns on JSC's control-flow profiler. A coverage report for a file is produced on demand by asking the profiler which basic blocks and functions of that file executed; --coverage-reporter=text prints the table to stderr, --coverage-reporter=lcov writes <coverage-dir>/lcov.info (via a temp file and rename).
  • CommandLineReporter is the single object that owns run-wide state for bun test (counts, JUnit buffer, timings); write_junit_report_if_needed and write_timings_if_needed were already the "must also run on bail" hooks, this adds coverage to that set.
  • The text reporter's "Uncovered Line #s" column drops a lone single-line range (pre-existing, unrelated, reported separately); the test fixture uses a two-line uncovered branch so its snapshot does not depend on that.
Notes

Rebased onto main after #39770, which turned the generate_code_coverage / print_code_coverage const generics into plain bool parameters. The conflict was in the helper and at the exec call site: write_code_coverage_if_needed now forwards opts.reporters.text, opts.reporters.lcov and Output::enable_ansi_colors_stderr() to the new signature instead of holding the old 8-way dispatch, and the exec block that did the same inline is replaced by the helper call as before. The &mut VirtualMachine to &VirtualMachine change and both bail sites applied unchanged. Re-ran the suites listed under Verified on the rebased build.

Rebased again after #39934, which appended a --parallel merges line coverage across workers test to the end of test/cli/test/coverage.test.ts, the same spot where the --bail block is added here. Kept both, main's test first. No source conflict. Re-ran the same suites.

Rebased a third time after #40678, which replaced the old generate_code_coverage / print_code_coverage / render_lcov trio with for_each_coverage_report plus a generate_code_coverage(vm, opts) that reads the reporter flags from opts and reports lcov write errors itself. That made the write_code_coverage_if_needed helper from earlier revisions redundant, so it is gone: bail_out now calls generate_code_coverage directly behind an enabled check, and the exec call site is left exactly as main has it. The &mut VirtualMachine to &VirtualMachine relaxation moved to the two new functions. The test file conflict was again a test appended at the end by main (#40586's --parallel merges function coverage across workers); kept both, main's first. Re-ran the same suites.

[review] gate passed · iteration 2 · 2 files touched

fails on main (without fix) 
ASAN without fix: 3 FAILED
$ BUN_DEBUG_QUIET_LOGS=1 bun scripts/build.ts --profile=debug --quiet test "--reporter=junit" "--reporter-outfile=/tmp/mechgate.xml" test/cli/test/coverage.test.ts
bun test v1.4.1 (65362b53b)

test/cli/test/coverage.test.ts:
bun test v1.4.1 (65362b53b)

demo.test.ts:
--------------|---------|---------|-------------------
File          | % Funcs | % Lines | Uncovered Line #s
--------------|---------|---------|-------------------
All files     |    0.00 |   66.67 |
 demo.test.ts |    0.00 |   66.67 | 1
--------------|---------|---------|-------------------

 0 pass
 0 fail
Ran 0 tests across 1 file. [212.00ms]
(pass) coverage crash [305.06ms]
bun test v1.4.1 (65362b53b)

demo2.ts:

 0 pass
 0 fail
Ran 0 tests across 1 file. [223.00ms]
(pass) lcov coverage reporter [291.87ms]
(pass) coverage excludes node_modules directory [268.41ms]
(pass) coveragePathIgnorePatterns - single pattern string [299.60ms]
(pass) coveragePathIgnorePatterns - partial coverage without nan [374.29ms]
(pass) coveragePathIgnorePatterns - array of patterns [312.20ms]
(pass) coveragePathIgnorePatterns - glob patterns [384.68ms]
(pass) coveragePathIgnorePatterns - lcov reporter [300.18ms]
(pass) co
... (truncated)

release without fix: 5 FAILED
bun test v1.4.1-canary.1 (65362b53b)

test/cli/test/coverage.test.ts:
bun test v1.4.1-canary.1 (65362b53b)

demo.test.ts:

 0 pass
 0 fail
Ran 0 tests across 1 file. [4.00ms]
(pass) coverage crash [7.80ms]
bun test v1.4.1-canary.1 (65362b53b)

demo2.ts:

 0 pass
 0 fail
Ran 0 tests across 1 file. [4.00ms]
(pass) lcov coverage reporter [8.04ms]
(pass) coverage excludes node_modules directory [6.81ms]
(pass) coveragePathIgnorePatterns - single pattern string [6.72ms]
184 | 
185 |   let stderr = result.stderr.toString("utf-8");
186 |   // Normalize output for cross-platform consistency
187 |   stderr = normalizeBunSnapshot(stderr, dir);
188 | 
189 |   expect(stderr).toMatchInlineSnapshot(`
                       ^
error: expect(received).toMatchInlineSnapshot(expected)

  
  "test.test.ts:
  (pass) should call only some functions
  ---------------|---------|---------|-------------------
  File           | % Funcs | % Lines | Uncovered Line #s
  ---------------|---------|---------|-------------------
  All files      |   75.00 |   83.33 |
-  include-me.ts |   50.00 |   66.67 | 6
+  include-me.ts |   50.00 |   66.67 | 
   test.test.ts  |  100.00 |  100.00 | 
  ----------
... (truncated)
passes on PR (with fix) 
ASAN with fix: all passed
$ BUN_DEBUG_QUIET_LOGS=1 bun scripts/build.ts --profile=debug --quiet test "--reporter=junit" "--reporter-outfile=/tmp/mechgate.xml" test/cli/test/coverage.test.ts
bun test v1.4.1 (65362b53b)

test/cli/test/coverage.test.ts:
bun test v1.4.1 (65362b53b)

demo.test.ts:
--------------|---------|---------|-------------------
File          | % Funcs | % Lines | Uncovered Line #s
--------------|---------|---------|-------------------
All files     |    0.00 |   66.67 |
 demo.test.ts |    0.00 |   66.67 | 1
--------------|---------|---------|-------------------

 0 pass
 0 fail
Ran 0 tests across 1 file. [208.00ms]
(pass) coverage crash [299.56ms]
bun test v1.4.1 (65362b53b)

demo2.ts:

 0 pass
 0 fail
Ran 0 tests across 1 file. [296.00ms]
(pass) lcov coverage reporter [365.33ms]
(pass) coverage excludes node_modules directory [327.84ms]
(pass) coveragePathIgnorePatterns - single pattern string [295.97ms]
(pass) coveragePathIgnorePatterns - partial coverage without nan [289.95ms]
(pass) coveragePathIgnorePatterns - array of patterns [298.49ms]
(pass) coveragePathIgnorePatterns - glob patterns [455.04ms]
(pass) coveragePathIgnorePatterns - lcov reporter [366.27ms]
(pass) co
... (truncated)

release with fix: all passed
$ bun scripts/build.ts --profile=release
[configured] bun-profile → bun (stripped) in 680ms (unchanged)
ninja: Entering directory `/workspace/bun/build/release'
[1/14] cxx obj/unified/UnifiedSource-src_jsc_bindings_node-0.cpp.o
[2/14] cxx obj/unified/UnifiedSource-src_uws_sys-0.cpp.o
[3/14] cxx obj/unified/UnifiedSource-src_jsc_bindings-5.cpp.o
[4/14] cxx obj/unified/UnifiedSource-src_jsc_bindings-4.cpp.o
[5/14] cxx obj/unified/UnifiedSource-src_jsc_bindings-0.cpp.o
[6/14] cxx obj/src/jsc/bindings/bindings.cpp.o
[7/14] cxx obj/unified/UnifiedSource-src_jsc_bindings-1.cpp.o
[8/14] cxx obj/unified/UnifiedSource-src_jsc_bindings-3.cpp.o
[9/14] gen generated_host_exports.rs
generated_host_exports.rs: 122 exports (host=5, lazy=10, generic=107, rust=0); 243 extern-C blocks audited
[10/14] gen cpp.rs (cppbind)
[10/14] cargo bun_runtime → libbun_runtime.a
   Compiling bun_runtime v0.0.0 (/workspace/bun/src/runtime)
    Finished `release` profile [optimized + debuginfo] target(s) in 4m 14s
[11/14] link bun-profile
[13/14] strip bun
[13/14] bun-profile --revision
1.4.1-canary.1+a08285685
[build] done
bun test v1.4.1-canary.1 (a08285685)

test/cli
... (truncated)
diff hotspot 
src/runtime/cli/test_command.rs |  59 ++++++++------
 test/cli/test/coverage.test.ts  | 176 +++++++++++++++++++++++++++++++++++++++-
 2 files changed, 208 insertions(+), 27 deletions(-)

gate history · 5 passed · 0 rejected · iteration 2

evidence per changed file 
file                             reads  edits  tests
src/runtime/cli/test_command.rs     22     17      0
test/cli/test/coverage.test.ts      10     10      0