fix(agent): say tap_at and hover_at act on the bare point where the engine can

Merged
SA
18d ago
+6-1

What and why

  • The built-in agent's rules told the model that every point tool acts on a listed control under the point by its id.
  • tap_at, hover_at, and drag_at (added in #1282) act on the bare points when the engine supports it, and fall back to the listed controls by id only when it does not (packages/e2e/src/agent/action-dispatcher.ts, executor.ts dragAt). type_at, press_at, and select_at do act on a listed control by id.
  • The rule in packages/e2e/src/agent/default-agent.ts now says both, matching docs/reference/agent.mdx and the tool descriptions.
  • Prompt text is not part of the replay cache key, so committed recordings stay valid.

Issue

Fixes #1367

Verified

Ran it locally: yes

  • The clause in main's source against the built package on this branch (packages/e2e/dist/agent/default-agent.js):
mainthis branch
a listed control under the point is acted on by its id.tap_at, hover_at, and drag_at act on the bare points where the engine can, and on the listed controls under them by id where it cannot; type_at, press_at, and select_at act on a listed control under the point by its id.
  • After merging main (e087ed2c4): pnpm check passed, e2e unit 3171 passed, 1 skipped.
  • pnpm --filter @e2e-dev/testbed test: 79 passed, 1 skipped.
  • No live agent run: there is no model key on this machine, so how a model reacts to the new wording is unchecked.

Checklist

  • Changeset added (pnpm changeset) for any change to a published package, or the change is testbed, docs, or CI only.
  • Breaking change: not breaking.
  • Docs updated in the same PR: not needed, docs/reference/agent.mdx and skills/e2e/ already describe this behavior.
  • Contract change: none.
  • Engine contract change (e2e/engine): none.