test: a test's time limit is its package's, for the author package and the verifier

Merged
AL
2d ago
+55-25

Closes #255.

CI on main failed after #254 merged: a test in packages/author timed out at 5011 ms on vitest's 5 s default. It is the third such failure in three days (5010, 5025 and 5011 ms), another test each time. After each one, the test that failed got a number of its own, and the next one failed.

What changes

  • packages/author/vitest.config.ts: the Node project sets testTimeout and hookTimeout to 120 s, as embed, mcp, the demo and the skills project already do for the whole package.
  • The author's tests lose the five numbers they carried (20 s twice, 60 s twice, 120 s once): each is lower than the package's limit, or the same.
  • packages/verify/vitest.config.ts is new, with the same limit. Every test there that plays a package starts a browser, and each heavy one carried its own number, so a new one written without a number would have failed on the default.
  • The verifier's tests lose their sixteen numbers of 120 s or less. The three that ask for 240 s stay, and so does { timeout: 1000 } on the test whose point is the time.
  • design/testing.md has the rule: a limit stops a hang and does not time a test, so it is set once for a package, for a slow runner.

Why this and not another number on one test

Over three CI runs, these are the author tests that take longest on a runner:

TestFastestSlowestIts limit before
The Course Presentation example, built, read and built again10.5 s13.1 s60 s, its own
The Interactive Video example, the same5.8 s7.5 s60 s, its own
bundle --all4.1 s5.4 s20 s, its own
An Interactive Video's time past the end4.1 s5.3 s20 s, its own
Every content type's bundle or none3.6 s5.0 s5 s, the default
A poster sized by what it holds2.3 s3.1 s5 s, the default

The last two had no number. The first of them failed this run, and the second is next.

Tested

  • Under load: bundle.test.ts and layout.test.ts with 30 busy processes on 10 cores, to stand in for a slow runner.
    • On main, the two tests with no number fail, at 5006 and 5014 ms.
    • On this branch all 33 pass.
  • The suites: the author package's 278 tests and the verifier's 29 pass, Chromium included.
  • Type check: passes in both packages.

Not changed

  • The player's unit project stays on the default. Two tests there carry 30 s of their own and run in under a second on a runner. Moving them would start the player's whole browser suite for this change.
  • cmi5, normalize, the runtime and the extension's tests run in milliseconds and keep the default. The rule says when a package gets a limit: when one of its tests first needs a number.

Summary by cubic

Moves test time limits from individual tests to the package config for packages/author and packages/verify, fixing CI failures from vitest's 5-second default on slow runners.

  • Sets testTimeout and hookTimeout to 120 s in both packages' vitest.config.ts, and to 180 s in the author's browser project.
  • Removes per-test timeouts of 120 s or less from the author's and verifier's tests; the three 240 s limits and the test measuring time stay.
  • Documents in design/testing.md that a package sets its own limit, and a test names one only to ask for more or where time is what it tests.