feat(e2e): add onAction hook to agent.act
Merged
OK
8d ago+425+425+012345678901234567890123456789-20-20-01234567890123456789
Adds onAction to agent.act: a hook that runs after each action the step completes, once the screen settles. Users asked for a screenshot after every agent action. Instead of a screenshot: true flag, this ships one general primitive: the screenshot is one line of user code, and the next request (logging, a live UI, metrics) needs no new API.
await agent.act('open settings and turn on dark mode', {
onAction: ({ index, action, source, summary }) => app.screenshot(`${index} ${action}`),
});
- It fires from the one point where every action commits,
ActionDispatcher.runActionNow(packages/e2e/src/agent/action-dispatcher.ts), so model and cache-replayed actions both reach it (source: 'model' | 'replay'). Mutating project tools reach it astool:<name>. - Before the hook runs, the screen settles the way the action's policy asks (
SETTLE_AFTER[kind].look), so a frame shows the effect and not the screen mid-transition. That capture is skipped when no hook is set. - A throw from the hook, or from that settle, hard-stops the step as an
ActionHookError. It never reaches the executor as a failed action to retry (which would double-tap a committed action) or the replay as a divergence. - An
ActionHookErrorconcludes the cache step asno-verdict, so a broken hook neither saves nor evicts a recording. - A verifying step that runs inside another step (an
expectin the hook) now proves only the steps before that one (packages/e2e/src/run/steps.ts). Before, it would have confirmed the act's half-checked recording. StepAccounting.checkpointnow throws once the step has hard-stopped, so a batched action queued behind a failed hook doesn't run.
flowchart LR
A[executor or replay] --> D[runActionNow]
D --> E[engine action]
E --> R[record trace]
R --> H{onAction set?}
H -- no --> N[next action]
H -- yes --> S[settle] --> U[hook] --> N
U -- throws --> X[ActionHookError: step fails, recording untouched]
Stacked on #1218, which keeps artifact screenshots after a secret fill. Without it, the example above fails on every sign-in act.
Not in this PR: ActionEvent.action is a string, not a typed union, so the grammar stays out of the public API for now. index is the action-budget slot, so a failed attempt skips a number.
Verified
Ran it locally: yes
- testbed, live model, the real CLI (
e2e run --config e2e.agent.config.ts):agent.act('sign in ...', { onAction })withapp.screenshot()in the hook. The first run gave 3 model calls and 3 frames taggedmodel. The cached rerun gave 0 model calls and the same 3 frames taggedreplay.
✓ ... > agent.act "sign in with the given username and password" 8.97s · 4 model calls
✓ ... > agent.act "sign in with the given username and password" 606ms
001-1-replay-type__type__admin__into_textbox__Username_.png
002-2-replay-typeSecret__fill_secret__admin.password__into_textbox__.png
003-3-replay-tap__tap_button__Sign_in_.png
| after the secret fill | after Sign in (settled) |
|---|---|