node:test: report a verdict for each suite registered with bun:test under run()

RO
3mo ago
+593-84

Problem

  • Under node:test's run(), a describe() / suite() block declared at the top level of a file (or nested in another one) never gets a test:pass / test:fail event of its own. Node v26.3.0 reports one after the suite's children. Only suites declared inside a running test (inline suites) were reported. Same on 1.4.0 and main; never worked.
  • Consequences for run() consumers: the suites N diagnostic and counts.suites undercount (the file in the test below reports suites 3 on main, node says suites 7), and a failure of the suite itself is invisible: a failing suite-level after() hook, or a describe() callback that throws, only surfaced as a synthesized test:fail for the whole file carrying the child's stderr, where node reports the suite as failed (failed running after hook, hookFailed) and nothing for the file.
  • Cause, in src/js/node/test.ts: addSuite() registers such a suite as a bun:test describe block and nothing reports it when the block is done; reportNodeToRunParent() only handles tests, and the suite event was emitted from scheduleSuiteSubtest() alone, the inline path. On the parent side, runOneFile() derived the per-file success and the "did the file itself die" check from counts.failed, which counts tests only, so a failed suite could not make a file unsuccessful and would not have stopped the file-level verdict either.

Fix

  • Child side, only when the file runs as a run() child (NODE_TEST_CONTEXT set): the describe callback wrapper (buildReportedSuite()) registers a bun:test beforeAll / afterAll pair in the block before running the callback. bun:test runs a block's beforeAll hooks before and its afterAll hooks after everything declared in the block, nested blocks included, so the beforeAll starts the suite's clock and the afterAll (settleReportedSuite()) runs the suite's after() hooks, stopping at the first failure like node's runHook(), and reports the verdict. Nested suites therefore report before their parent, and children before their suite, which is node's order.
  • The verdict is node's Suite.run() + Test.postRun(), in concludeSuite(): a failure recorded against the suite itself wins (a before() / after() hook failure, recorded as failed running <kind> hook / hookFailed with the hook's error as cause, or a throwing / rejecting describe callback), otherwise children that finished without passing and are not todo fail it with N subtests failed / subtestsFailed. rollUpIntoEnclosingSuite() is the roll-up for tests and suites registered with bun:test; inline subtests keep theirs. Under run(), after() hooks of such a suite are collected on the node for the closing hook to run (their failure still reaches bun:test through the hook's done(err), exactly what after() handed it before, so the child's exit status is unchanged); before() hooks keep registering with bun:test and additionally record their failure on the suite.
  • A todo suite's own failures are advisory in node (countCompletedTest leaves success alone, the process exits 0): they are recorded so the suite reports test:fail with the todo directive, and not handed to bun:test. The same path now reports todo suites in general, which were previously reported as passed at declaration time, before their children; skipped suites are unchanged (reportSkippedNode(), formerly reportDirectiveOnlyNode(), is now only ever called for skips).
  • A describe callback that throws or rejects is reported at declaration time and rethrown: bun:test drops the whole block in that case, hooks included, so this is the one verdict that cannot be emitted in execution order. The inline suite path (scheduleSuiteSubtest()) now goes through the same concludeSuite() / reportSuiteToRunParent(), so both kinds of suite produce the same event shape; its only visible changes are a real duration_ms instead of 0 and the first failure attached as the cause of subtestsFailed, which executeTestNode() already did for tests (makeSubtestsFailedError() is the shared helper).
  • Parent side: runFiles() / runOneFile() carry node's harness.success next to the counters (RunTally). republishChildEvent() clears it for a failed or cancelled test and for a failed non-todo suite, as node's countCompletedTest() does; the per-file and run-level test:summary report it, and the file-level verdict is now synthesized only for a nonzero exit that no reported failure accounts for (previously: no failed test), so a suite-only failure yields success: false, fail 0, and no file-level event, as in node.
  • Why this is correct: it matches node. For every fixture in the new tests, node v26.3.0 and this branch agree on the sequence of verdicts (names, details.type, directives, error messages and failure types), on the suites diagnostic and on every test:summary field the change touches; the comparison is in the details block. The known remaining differences are listed there too; none of them changes a verdict or a count.
  • Both new hooks declare and call done, like every other callback this module hands to bun:test: bun:test derives "waits for done()" from the callback's arity, and cannot read it through the async-context wrapper a callback registered under an active AsyncLocalStorage context gets, which is the case for every nested describe callback (it runs inside its parent's runWithNode()). A zero-arity hook there waits 5s for a done() nobody calls and, for a beforeAll, skips the block's tests. #38910 fixes that in bun:test; the code here is right either way.
  • Plain bun test on a node:test file is untouched: every new code path is behind runChildReporterEnabled, and rollUpIntoEnclosingSuite() returns immediately outside a run() child.
  • Verified: test/js/node/test_runner/node-test.test.ts, new node:test run() block (5 cases: the suite sequence above, suite-only after-hook failure, todo suites' own failures, before-hook failure, throwing and rejecting describe callbacks). All 5 fail on the unfixed build (no suite verdicts, suites 3 instead of 7, file-level verdicts instead of suite ones) and pass with it; the 45 existing cases in the file still pass. The ported test/js/node/test/parallel/test-runner-* files behave as on main (test-runner-todo-skip-tests.js, which pins a skipped suite's single event, included; test-runner-mock-timers-scheduler.js fails its 100ms wall-clock bound on the unmodified debug build too). test/regression/issue/{19111,19412,24147,24338,24339,24374,26915,28431}.test.ts pass.

Background

  • run() is node:test's programmatic runner. With the default process isolation it runs each file in a child process; the child reports its tests as events, and the parent republishes them on the stream run() returned and appends the run-level plan, diagnostics and test:summary. Bun's run() spawns bun test on each file with NODE_TEST_CONTEXT set; in that mode node:test prints one JSON event per verdict, which the parent rebuilds.
  • In node every suite is itself reported: after its children, as a test:pass or test:fail whose details.type is "suite". countCompletedTest() (node's test_runner/utils.js) counts such an event only in suites, never in tests / passed / failed, and clears the harness's success flag when the suite failed and is not todo. That flag is what test:summary reports as success; it is the only place a suite failure shows up in the totals.
  • Node's FileTest synthesizes a verdict for the file itself only when the child exited nonzero without any of its own top-level verdicts explaining it (a file that failed to load, a root-level hook failure); a file whose tests or suites failed is represented by those verdicts.
  • Bun's node:test registers top-level tests and suites with bun:test (Bun.jest), which then drives execution: tests through a runner callback, suites as describe blocks. Suites declared while a test is running are executed by the module itself ("inline" suites). bun:test orders a block's hooks around its contents, so a beforeAll / afterAll pair registered from the describe callback observes the start and the end of everything in the block.
node v26.3.0 vs. this branch on the test fixtures, and the remaining differences

Verdicts as name (details.type, directive), in stream order, for suites.js from the first test. The left column is node v26.3.0 and is also, line for line, what this branch emits:

node v26.3.0 = this branch                     bun main
skipped suite (suite, skip)                    skipped suite (suite, skip)
ok                                             todo suite (suite, todo)
bad FAIL                                       ok
inner (suite) FAIL "1 subtest failed"          bad FAIL
ok2                                            ok2
outer (suite) FAIL "1 subtest failed"          fails inside a todo suite FAIL (todo)
empty suite (suite)                            todo child FAIL (todo)
fails inside a todo suite FAIL (todo)          skipped child (skip)
todo suite (suite, todo)                       inline child
todo child FAIL (todo)                         inline suite (suite)
skipped child (skip)                           parent test
suite with directives (suite)
inline child
inline suite (suite)
parent test
suites 7                                       suites 3

after-hook.js (one suite whose after() throws, one passing child):

node: ok; suite (suite) FAIL "failed running after hook" hookFailed, cause "after boom"; summary success false, fail 0, suites 1
main: ok; after-hook.js FAIL (file-level, error = the child's stderr); summary success false, fail 1, suites 0
this: same as node

todo-suites.js (todo suites whose before(), after(), and callback fail): node and this branch both report the three suites as test:fail with the todo directive and the hook / callback error, success: true, todo 3, suites 3, no file-level verdict. Main reported the suites as passed at declaration time plus a file-level failure.

Remaining differences, all pre-existing or noted in the code:

  • Children of a suite whose before() hook or callback failed: node cancels them (cancelledByParent); here bun:test decides what happens to them. After a failed before() they currently run and pass (bun:test only prints an error passed to a hook's done(); once it fails the hook instead, #39112, it will skip them), so the before-hook test only pins the suite's verdict and the summary fields that hold either way. After a throwing callback bun:test drops them (no events), except in a todo suite, where the error is not rethrown and they run; there the counts agree with node, which counts the cancelled children as todo as well.
  • A suite whose callback threw is reported when it is declared (bun:test drops the block), so it appears earlier in the stream than in node. Skipped suites were already reported at declaration time; unchanged.
  • Shape: the child does not label test code failures testCodeFailure (pre-existing for tests, same for describe callbacks here), and subtestsFailed errors carry the first failure as cause where node repeats the message. nesting / testNumber / topLevel are #39284's.
  • Inline suites still fold their own hook or callback failure into N subtests failed (pre-existing); the shared concludeSuite() would take a recorded error there too.