feat(cli): add --action-timeout and --assertion-timeout to e2e run

Merged
KA
6d ago
+149-8

Adds --action-timeout <ms> and --assertion-timeout <ms> to e2e run, so a test you're still writing fails in a few seconds instead of waiting out the 30 s action timeout.

npx e2e run tests/signup.e2e.ts --action-timeout 5000
  • Each flag replaces the config's actionTimeout or assertionTimeout for one run. They go into the same CliOverrides as --retries (packages/e2e/src/config/resolve.ts), so a child-process worker that resolves the config again gets them through its bootstrap (packages/e2e/src/run/worker/entry.ts:159).
  • A call's own { timeout } still wins, as in Playwright. The code already does this: timeoutMs ?? this.options.actionTimeout (packages/e2e/src/locator/engine.ts:119), timeout ?? engine.assertionTimeout (packages/e2e/src/expect/async.ts:172, expect/poll.ts:92), timeout ?? environment.config.actionTimeout (run/fixture-recording.ts:80). The integration test covers it, and the docs now say it.
  • Validation: the parser takes an integer 1-2147483647, the --max-duration bounds. Anything else is a usage error, exit 2. resolveConfig checks again for the run() path with timerMs, which returns INVALID_CONFIG, also exit 2.
  • Watch mode keeps the flags on every rerun, since planRun spreads the base options (packages/e2e/src/cli/watch.ts:57).
  • Docs: rows in docs/reference/cli.mdx, override and precedence bullets under actionTimeout and assertionTimeout in docs/reference/config.mdx, the flag row in skills/e2e/references/running.md, and an authoring tip in the Tools table of skills/e2e/references/debugging.md.

Decisions

  • Not on e2e explore or e2e mcp. e2e explore --timeout is the exploration's wall clock (180000-900000, explore/index.ts:33), not the config's timeout, and an exploration has no test lines to fail fast on. An MCP session's actions target node ids from the last observe, so they don't wait out the action timeout the way a wrong locator does. e2e run still has no --timeout. The new names follow --max-duration -> maxDuration: config key in kebab case.
  • The config value is still checked when a flag replaces it. --retries short-circuits (boundedInt(cli) ?? boundedInt(raw)), so retries: 99 passes while --retries 1 is given. Here actionTimeout: '30s' fails with or without the flag.
  • Upper bound. A fresh-context review caught that setTimeout fires after 1 ms for anything above 2147483647. Without the bound, a fixture call such as browser.evaluate or session.save() would fail almost at once under --action-timeout 999999999999. The flags now stop at the timer maximum. The config keys (actionTimeout, assertionTimeout, and timeout, launchTimeout, cleanupTimeout) already had this gap. I left them alone so this PR doesn't change what configs are accepted.
  • No wire change. report.json records no timeouts. The config digest covers the config file only, so a flag can't trip the worker's digest check. The replay cache key doesn't include the digest, so recordings stay valid under the flags.

Verified

Ran it locally: yes

Built CLI on the A/B harness scratch project (run-arm.mjs --setup-only --seed seeds/fix-journey). The seeded test has a wrong label on /forms. Main is the base commit's build (t3-6e7ffaee, d7c6b5e5).

mainthis branch
e2e run (wrong label)fails at getByLabel("Name") after 30.46 s, process 31.2 ssame, 30.48 s
e2e run --action-timeout 3000error: unknown option '--action-timeout', exit 2fails at the same line after 3.37 s, process 4.2 s, same nearest nodes and trace page
fixed test, --action-timeout 3000 --assertion-timeout 2000-passes, 681 ms
wrong summary text, e2e run5.73 s5.73 s
wrong summary text, --assertion-timeout 1000unknown optionASSERTION_FAILED after 1.76 s
$ node node_modules/e2e/dist/cli/bin.js run --action-timeout 3000
 ✓ target "web" command ready 87ms
 ❯ |web| tests/journey.e2e.ts (1 test | 1 failed) 3.37s
   × an admin signs in, saves a profile, and creates a workspace 3.37s
     → locator matched no nodes within getByLabel("Name")
...
 ❯ tests/journey.e2e.ts:14:3
 ❯ on screen #n12032 textbox "Full name" purpose=username
 ❯ trace .e2e/results/journey-an-admin-signs-in-saves-a-d41c531a416735aa/trace.md

$ node node_modules/e2e/dist/cli/bin.js run --action-timeout 0
error: option '--action-timeout <ms>' argument '0' is invalid. must be an integer from 1 through 2147483647
$ echo $?
2
  • Testbed default suite on a port-0 copy of its config (another run held port 4271), --action-timeout 5000 --assertion-timeout 5000: 15 files, 79 passed, 1 skipped, 18.0 s. Without the flags: the same results, 17.1 s.
  • The new integration test (tests/integration/cli-run.test.ts) runs the built CLI on the fake engine with 30 s timeouts in the config. With the runner line that forwards --action-timeout deleted, the wrong-locator step takes 30,023 ms and the test fails. With it, the step takes about 400 ms.

A/B, fix task (n=2, sanity check)

run-arm.mjs --task tasks/fix-journey.md --seed seeds/fix-journey, Sonnet 5.5, main's skill against this branch's.

RunWall mine2e runsSessionsMCP callsTool callsModel stepsCost $Final exit
main-fix-11.740014120.100
main-fix-21.120014110.060
f3-fix-11.64001480.120
f3-fix-21.02001190.120
  • Both branch agents read references/debugging.md before their first run. Neither used the flag on that run, so both paid the 30 s timeout once.
  • f3-fix-1 had two 30 s runs, then ran npx e2e run tests/journey.e2e.ts --action-timeout 5000 for the third. That run hit the assertion mistake, so it took 5.7 s whatever the flag.
  • f3-fix-2 read the app's source through the server path in e2e.config.ts and fixed all three mistakes after one run.
  • Nothing here shows a speedup. It shows the tip gets read and picked up, late. Putting it in SKILL.md's run step might get it onto the first run, but that line belongs to the rerun-loop PR in this batch, so I kept to debugging.md.