fix(agent): say tap_at and hover_at act on the bare point where the engine can
Merged
SA
18d ago+6+6+0123456789-1-1-0123456789
What and why
- The built-in agent's rules told the model that every point tool acts on a listed control under the point by its id.
tap_at,hover_at, anddrag_at(added in #1282) act on the bare points when the engine supports it, and fall back to the listed controls by id only when it does not (packages/e2e/src/agent/action-dispatcher.ts,executor.tsdragAt).type_at,press_at, andselect_atdo act on a listed control by id.- The rule in
packages/e2e/src/agent/default-agent.tsnow says both, matchingdocs/reference/agent.mdxand the tool descriptions. - Prompt text is not part of the replay cache key, so committed recordings stay valid.
Issue
Fixes #1367
Verified
Ran it locally: yes
- The clause in main's source against the built package on this branch (
packages/e2e/dist/agent/default-agent.js):
| main | this branch |
|---|---|
| a listed control under the point is acted on by its id. | tap_at, hover_at, and drag_at act on the bare points where the engine can, and on the listed controls under them by id where it cannot; type_at, press_at, and select_at act on a listed control under the point by its id. |
- After merging main (e087ed2c4):
pnpm checkpassed, e2e unit 3171 passed, 1 skipped. pnpm --filter @e2e-dev/testbed test: 79 passed, 1 skipped.- No live agent run: there is no model key on this machine, so how a model reacts to the new wording is unchecked.
Checklist
- Changeset added (
pnpm changeset) for any change to a published package, or the change is testbed, docs, or CI only. - Breaking change: not breaking.
- Docs updated in the same PR: not needed,
docs/reference/agent.mdxandskills/e2e/already describe this behavior. - Contract change: none.
- Engine contract change (
e2e/engine): none.