feat(cli): add --action-timeout and --assertion-timeout to e2e run
Merged
KA
6d ago+149+149+012345678901234567890123456789-8-8-0123456789
Adds --action-timeout <ms> and --assertion-timeout <ms> to e2e run, so a test you're still writing fails in a few seconds instead of waiting out the 30 s action timeout.
npx e2e run tests/signup.e2e.ts --action-timeout 5000
- Each flag replaces the config's
actionTimeoutorassertionTimeoutfor one run. They go into the sameCliOverridesas--retries(packages/e2e/src/config/resolve.ts), so a child-process worker that resolves the config again gets them through its bootstrap (packages/e2e/src/run/worker/entry.ts:159). - A call's own
{ timeout }still wins, as in Playwright. The code already does this:timeoutMs ?? this.options.actionTimeout(packages/e2e/src/locator/engine.ts:119),timeout ?? engine.assertionTimeout(packages/e2e/src/expect/async.ts:172,expect/poll.ts:92),timeout ?? environment.config.actionTimeout(run/fixture-recording.ts:80). The integration test covers it, and the docs now say it. - Validation: the parser takes an integer 1-2147483647, the
--max-durationbounds. Anything else is a usage error, exit 2.resolveConfigchecks again for therun()path withtimerMs, which returnsINVALID_CONFIG, also exit 2. - Watch mode keeps the flags on every rerun, since
planRunspreads the base options (packages/e2e/src/cli/watch.ts:57). - Docs: rows in
docs/reference/cli.mdx, override and precedence bullets underactionTimeoutandassertionTimeoutindocs/reference/config.mdx, the flag row inskills/e2e/references/running.md, and an authoring tip in the Tools table ofskills/e2e/references/debugging.md.
Decisions
- Not on
e2e exploreore2e mcp.e2e explore --timeoutis the exploration's wall clock (180000-900000,explore/index.ts:33), not the config'stimeout, and an exploration has no test lines to fail fast on. An MCP session's actions target node ids from the lastobserve, so they don't wait out the action timeout the way a wrong locator does.e2e runstill has no--timeout. The new names follow--max-duration->maxDuration: config key in kebab case. - The config value is still checked when a flag replaces it.
--retriesshort-circuits (boundedInt(cli) ?? boundedInt(raw)), soretries: 99passes while--retries 1is given. HereactionTimeout: '30s'fails with or without the flag. - Upper bound. A fresh-context review caught that
setTimeoutfires after 1 ms for anything above 2147483647. Without the bound, a fixture call such asbrowser.evaluateorsession.save()would fail almost at once under--action-timeout 999999999999. The flags now stop at the timer maximum. The config keys (actionTimeout,assertionTimeout, andtimeout,launchTimeout,cleanupTimeout) already had this gap. I left them alone so this PR doesn't change what configs are accepted. - No wire change.
report.jsonrecords no timeouts. The config digest covers the config file only, so a flag can't trip the worker's digest check. The replay cache key doesn't include the digest, so recordings stay valid under the flags.
Verified
Ran it locally: yes
Built CLI on the A/B harness scratch project (run-arm.mjs --setup-only --seed seeds/fix-journey). The seeded test has a wrong label on /forms. Main is the base commit's build (t3-6e7ffaee, d7c6b5e5).
| main | this branch | |
|---|---|---|
e2e run (wrong label) | fails at getByLabel("Name") after 30.46 s, process 31.2 s | same, 30.48 s |
e2e run --action-timeout 3000 | error: unknown option '--action-timeout', exit 2 | fails at the same line after 3.37 s, process 4.2 s, same nearest nodes and trace page |
fixed test, --action-timeout 3000 --assertion-timeout 2000 | - | passes, 681 ms |
wrong summary text, e2e run | 5.73 s | 5.73 s |
wrong summary text, --assertion-timeout 1000 | unknown option | ASSERTION_FAILED after 1.76 s |
$ node node_modules/e2e/dist/cli/bin.js run --action-timeout 3000
✓ target "web" command ready 87ms
❯ |web| tests/journey.e2e.ts (1 test | 1 failed) 3.37s
× an admin signs in, saves a profile, and creates a workspace 3.37s
→ locator matched no nodes within getByLabel("Name")
...
❯ tests/journey.e2e.ts:14:3
❯ on screen #n12032 textbox "Full name" purpose=username
❯ trace .e2e/results/journey-an-admin-signs-in-saves-a-d41c531a416735aa/trace.md
$ node node_modules/e2e/dist/cli/bin.js run --action-timeout 0
error: option '--action-timeout <ms>' argument '0' is invalid. must be an integer from 1 through 2147483647
$ echo $?
2
- Testbed default suite on a port-0 copy of its config (another run held port 4271),
--action-timeout 5000 --assertion-timeout 5000: 15 files, 79 passed, 1 skipped, 18.0 s. Without the flags: the same results, 17.1 s. - The new integration test (
tests/integration/cli-run.test.ts) runs the built CLI on the fake engine with 30 s timeouts in the config. With the runner line that forwards--action-timeoutdeleted, the wrong-locator step takes 30,023 ms and the test fails. With it, the step takes about 400 ms.
A/B, fix task (n=2, sanity check)
run-arm.mjs --task tasks/fix-journey.md --seed seeds/fix-journey, Sonnet 5.5, main's skill against this branch's.
| Run | Wall min | e2e runs | Sessions | MCP calls | Tool calls | Model steps | Cost $ | Final exit |
|---|---|---|---|---|---|---|---|---|
| main-fix-1 | 1.7 | 4 | 0 | 0 | 14 | 12 | 0.10 | 0 |
| main-fix-2 | 1.1 | 2 | 0 | 0 | 14 | 11 | 0.06 | 0 |
| f3-fix-1 | 1.6 | 4 | 0 | 0 | 14 | 8 | 0.12 | 0 |
| f3-fix-2 | 1.0 | 2 | 0 | 0 | 11 | 9 | 0.12 | 0 |
- Both branch agents read
references/debugging.mdbefore their first run. Neither used the flag on that run, so both paid the 30 s timeout once. - f3-fix-1 had two 30 s runs, then ran
npx e2e run tests/journey.e2e.ts --action-timeout 5000for the third. That run hit the assertion mistake, so it took 5.7 s whatever the flag. - f3-fix-2 read the app's source through the server path in
e2e.config.tsand fixed all three mistakes after one run. - Nothing here shows a speedup. It shows the tip gets read and picked up, late. Putting it in
SKILL.md's run step might get it onto the first run, but that line belongs to the rerun-loop PR in this batch, so I kept todebugging.md.