feat(agent): add drag_at to the pixel tools

Merged
MA
18d ago
+1190-562

What and why

  • The agent has tap_at, hover_at, type_at, press_at, and select_at, but no drag, so agent.act cannot move a shape on a canvas or a pin on a map that the tree does not list. Engines already declare dragTo in pointerActions with a viewport point target; only the agent side was missing. Split out of #1277.
  • New agent tool drag_at {x, y, to_x, to_y}, in screenshot pixels like the other point tools. New executor action ctx.actions.dragAt(from, to), which returns a PointDragResult.
  • Same rule as tap_at after #1284: the bare points go to the engine's dragTo pointer kind, where the model aimed. An engine without that kind drags by id when the points land on two different listed controls, as an ordinary drag, and fails with UNSUPPORTED_CAPABILITY otherwise (a node dragged onto itself does nothing).
  • The trace records a dragAt entry as from: { point, within? } and to: { point, within? } plus the viewport. Each within is the point's place inside the listed node that contained it, with that node's container and place among twins, so each end follows a layout shift on replay the way a tap_at point does. One reader and one text mapper serve tapAt/hoverAt and both drag ends. In replay.ts the planned point and within kinds are now one point kind with a PointEnd, which the drag reuses for both ends.
  • Additive only: ExecutorActions gains a member and no existing signature changes. GrammarVerb is internal, so the engine contract (e2e/engine) is unchanged.
flowchart LR
  A["drag_at(x, y, to_x, to_y)"] --> B{"engine declares<br/>pointerActions dragTo?"}
  B -- yes --> C["performAt(from, { kind: 'dragTo', target: to })"]
  B -- no --> D{"ends on two different<br/>listed controls?"}
  D -- yes --> E["drag by id"]
  D -- no --> F["UNSUPPORTED_CAPABILITY"]

From a custom StepExecutor:

const result = await ctx.actions.dragAt({ x: 180, y: 36 }, { x: 120, y: 60 });
// result.source and result.destination name the listed controls at each end, when there are any.

Code: packages/e2e/src/agent/action-dispatcher.ts (pointDrag), packages/e2e/src/agent/primitives.ts (drag_at), packages/e2e/src/cache/trace.ts (PointDragAction), packages/e2e/src/agent/replay.ts (pointDrag plan).

Verified

Ran it locally: yes

  • packages/e2e/tests/integration/agent-act-pixel-tools.test.ts with real Chromium on the canvas fixture. A scripted agent calls drag_at {x: 180, y: 36, to_x: 120, to_y: 60} on a pin the tree does not list, and the page logs moved red to 200,100. A second case drags between two listed controls on the web engine, which gets the bare points (drag from the point (50, 275) on button "Reset" to the point (100, 315) on textbox "Note"). A fixture engine with node drags only joins the same kind of two controls by id and refuses two points inside one control with UNSUPPORTED_CAPABILITY.
  • 2026-10-28, merged main d2ffb319c: resolved the mcp/catalog.ts doc comment (main added run_code), the web MCP catalog count is now 29, and the performAt JSDoc names drag_at. pnpm check and the full e2e suite (3993 tests) pass. Dropping the to end's placement in the recorder fails 2 recorder tests; routing by id first fails the bare-points web case.

Checklist

  • Changeset added (pnpm changeset) for any change to a published package, or the change is testbed, docs, or CI only.
  • Breaking change: title carries !, the changeset body names what breaks and what replaces it, and the deprecation warning landed a window earlier (stability policy in CONTRIBUTING.md). Not breaking.
  • Docs updated in the same PR: the docs/**/*.mdx page for the behavior, and skills/e2e/ if the skill describes it.
  • Contract change: emitted .d.ts reviewed, tests/types/sdk-types.ts updated; wire change edits the schema and both fixtures. No wire change.
  • Engine contract change (e2e/engine): changesets for e2e, @e2e-dev/web, and @e2e-dev/mobile. Not an engine contract change.

🤖 Generated with Claude Code