bun test: fix inline snapshot conflict, beforeAll skip, and test.each title reporting

RO
2mo ago
+273-52

Three bun test reporter correctness bugs. Each is deterministic, each has a test that fails on current main and passes with the fix.

1. A failing inline snapshot reports as passing

import { test, expect } from "bun:test";
test("t", () => { for (const v of [1, 2]) expect(v).toMatchInlineSnapshot(); });

On current main this prints error: Failed to update inline snapshot: Multiple inline snapshots on the same line must all have the same value, exits 1, and never modifies the source file, while the reporter says 1 pass / 0 fail / snapshots: +1 added and the JUnit report says tests="1" failures="0". CI keyed on JUnit or test summaries shows green for a run whose exit code is red, and the failing test is recorded as passing.

Cause: the conflict is only detected by Snapshots::write_inline_snapshots, which runs after the whole test run, when it can no longer be attributed to a test. Expect::inline_snapshot pushes onto inline_snapshots_to_write blindly and returns Ok(undefined), so the test is marked Pass.

Fix: check the pending queue for an existing entry at the same (line, col) at record time. If a different value is already queued, throw from the matcher, so the test itself fails and the summary, the JUnit report, and the exit code all agree. The queued entries for that call site are removed so the source file is still left untouched.

2. Tests vanish when a beforeAll hook throws

import { test, beforeAll } from "bun:test";
beforeAll(() => { throw new Error("boom") });
test("a", () => {});
test("b", () => {});

On current main this reports one (fail) (unnamed) and Ran 1 test across 1 file. Tests a and b are absent from the console output, the summary counts, and the JUnit report. docs/test/lifecycle.mdx says "all tests in this scope will be skipped".

Cause: on a beforeAll failure, step_group jumps group_index straight to failure_skip_to, so the jumped-over sequences never reach on_sequence_completed and never hit the reporter.

Fix: before taking the jump, report each jumped-over test sequence as skipped (or todo, for test.todo). Only hook groups have failure_skip_to > group_index + 1 (Order.rs), so the loop is a no-op for ordinary failing test groups, and hook sequences in the range (a second beforeAll) are not reported as testcases. Execution order is unchanged; failure-skip.test.ts's existing ordering snapshots all still pass.

3. test.each title formatting

test.each([[1.5]])("i %i", () => {});          // title "i %i"   (not substituted)
test.each([[1.5]])("s %s", () => {});          // title "s %s"   (not substituted)
test.each([[{ a: 1 }]])("p %p", () => {});     // title "p {\n  a: 1,\n}"  (multi-line)

%s only substituted string arguments and %i only integral numbers; any other argument left the literal %s/%i in the test name. %p (and $variable for object values) printed the value through the multi-line formatter, producing multi-line test names that corrupt the console layout and break JUnit consumers.

jest-each delegates these to node's util.format, which Bun already implements with the expected results (%s of 1.5 is "1.5", %i of 1.5 is "1"). Fix: route non-string %s arguments through the inspector on a single line, truncate non-integral numbers for %i, and format %p/$variable/%s objects on a single line. The never-written test.todo("check formatting for %p") in test/cli/test/bun-test.test.ts is implemented alongside the other specifier tests.

Also

grow file for new snapshot in snapshot.test.ts is registered via a bare test() without the isDebug ? 100_000 : 5_000 timeout that SnapshotTester.test applies. It does the most child spawns of any test in the file (4 update() calls), so under a debug build it times out at the default 5s, and the timed-out body's in-flight toMatchSnapshot() then lands on whichever test runs next, silently appending a bogus backtick in test name 2 entry to the file's own committed snapshot. It now gets the same timeout the SnapshotTester.test registrations have.

A review comment caught that the catch-all in the beforeAll skip-reporting loop mapped ScopeMode::FilteredOut to Result::Skip, where the normal path (step_sequence_one) maps it to Result::SkippedBecauseLabel. With a -t filter and a throwing beforeAll in the same scope, a filtered-out test was printed as a regular skip and counted in summary.skip instead of staying suppressed. Fixed with a ScopeMode::FilteredOut => Result::SkippedBecauseLabel arm and covered by a failing beforeAll keeps -t filtered-out tests suppressed.

test/regression/issue/12782.test.ts snapshots the reporter output for a fixture with a throwing beforeAll, so its inline snapshot is updated for the two tests that are now reported as skipped (2 skip, Ran 5 tests). Execution is unchanged; the fixture's tests still do not run. This was the only test in the repository whose expectation captures a failing-hook reporter line, and the only real failure in the CI run at f5dd9fa.

Verification

Each test fails on the released binary and passes with the fix:

  • test/js/bun/test/snapshot-tests/snapshots/snapshot.test.ts should error trying to update the same line twice: strengthened from an error-string check (which already passed, because the error string existed, just not attributed to the test) to assert (fail) whoops, 0 pass, 1 fail, failures="1" in JUnit, and that the source file is unchanged.
  • test/js/bun/test/failure-skip.test.ts a failing beforeAll reports its tests as skipped: asserts both tests appear as (skip), 2 skip, Ran 3 tests, and tests="3" failures="1" skipped="2" in JUnit.
  • test/js/bun/test/jest-each.test.ts title specifiers substitute every number and stay on one line: asserts the exact (pass) line for each specifier.
  • test/cli/test/bun-test.test.ts check formatting for %p.
  • test/js/bun/test/failure-skip.test.ts a failing beforeAll keeps -t filtered-out tests suppressed: asserts the -t filtered-out test never prints a (skip) line and is not counted (1 skip, Ran 2 tests). Fails on main (the test vanishes entirely) and without the FilteredOut arm (it shows as a regular skip, 2 skip, Ran 3 tests).

Also ran test/js/bun/test/{test-test,bun_test,jest-hooks,dots,concurrent,concurrent_immediate,only-failures,describe,test-failing,expect}.test.*, test/js/junit-reporter/junit.test.js, and test/cli/test/{bun-test,isolation}.test.ts against the debug build; no regressions. No committed .snap file anywhere in test/ has a multi-line key, so the %p title change cannot orphan an existing snapshot.