feat(agent): add drag_at to the pixel tools
Merged
MA
18d ago+1190+1190+0123456789012345678901234567890123456789-562-562-012345678901234567890123456789
What and why
- The agent has
tap_at,hover_at,type_at,press_at, andselect_at, but no drag, soagent.actcannot move a shape on a canvas or a pin on a map that the tree does not list. Engines already declaredragToinpointerActionswith a viewport point target; only the agent side was missing. Split out of #1277. - New agent tool
drag_at {x, y, to_x, to_y}, in screenshot pixels like the other point tools. New executor actionctx.actions.dragAt(from, to), which returns aPointDragResult. - Same rule as
tap_atafter #1284: the bare points go to the engine'sdragTopointer kind, where the model aimed. An engine without that kind drags by id when the points land on two different listed controls, as an ordinarydrag, and fails withUNSUPPORTED_CAPABILITYotherwise (a node dragged onto itself does nothing). - The trace records a
dragAtentry asfrom: { point, within? }andto: { point, within? }plus the viewport. Eachwithinis the point's place inside the listed node that contained it, with that node's container and place among twins, so each end follows a layout shift on replay the way atap_atpoint does. One reader and one text mapper servetapAt/hoverAtand both drag ends. Inreplay.tsthe plannedpointandwithinkinds are now onepointkind with aPointEnd, which the drag reuses for both ends. - Additive only:
ExecutorActionsgains a member and no existing signature changes.GrammarVerbis internal, so the engine contract (e2e/engine) is unchanged.
flowchart LR
A["drag_at(x, y, to_x, to_y)"] --> B{"engine declares<br/>pointerActions dragTo?"}
B -- yes --> C["performAt(from, { kind: 'dragTo', target: to })"]
B -- no --> D{"ends on two different<br/>listed controls?"}
D -- yes --> E["drag by id"]
D -- no --> F["UNSUPPORTED_CAPABILITY"]
From a custom StepExecutor:
const result = await ctx.actions.dragAt({ x: 180, y: 36 }, { x: 120, y: 60 });
// result.source and result.destination name the listed controls at each end, when there are any.
Code: packages/e2e/src/agent/action-dispatcher.ts (pointDrag), packages/e2e/src/agent/primitives.ts (drag_at), packages/e2e/src/cache/trace.ts (PointDragAction), packages/e2e/src/agent/replay.ts (pointDrag plan).
Verified
Ran it locally: yes
packages/e2e/tests/integration/agent-act-pixel-tools.test.tswith real Chromium on the canvas fixture. A scripted agent callsdrag_at {x: 180, y: 36, to_x: 120, to_y: 60}on a pin the tree does not list, and the page logsmoved red to 200,100. A second case drags between two listed controls on the web engine, which gets the bare points (drag from the point (50, 275) on button "Reset" to the point (100, 315) on textbox "Note"). A fixture engine with node drags only joins the same kind of two controls by id and refuses two points inside one control withUNSUPPORTED_CAPABILITY.- 2026-10-28, merged main d2ffb319c: resolved the
mcp/catalog.tsdoc comment (main addedrun_code), the web MCP catalog count is now 29, and theperformAtJSDoc namesdrag_at.pnpm checkand the fulle2esuite (3993 tests) pass. Dropping thetoend's placement in the recorder fails 2 recorder tests; routing by id first fails the bare-points web case.
Checklist
- Changeset added (
pnpm changeset) for any change to a published package, or the change is testbed, docs, or CI only. - Breaking change: title carries
!, the changeset body names what breaks and what replaces it, and the deprecation warning landed a window earlier (stability policy in CONTRIBUTING.md). Not breaking. - Docs updated in the same PR: the
docs/**/*.mdxpage for the behavior, andskills/e2e/if the skill describes it. - Contract change: emitted
.d.tsreviewed,tests/types/sdk-types.tsupdated; wire change edits the schema and both fixtures. No wire change. - Engine contract change (
e2e/engine): changesets fore2e,@e2e-dev/web, and@e2e-dev/mobile. Not an engine contract change.
🤖 Generated with Claude Code