fix(agent): tap_at and hover_at tap where the model aimed

Merged
KR
4d ago
+333-100

tap_at outside a native iOS popover never dismissed it. iOS lists a full-screen dismiss popup button behind the popover, so the point snapped to that button and the tap went by id to its center, which is inside the popover. The tap hit the calendar instead of the backdrop.

  • tap_at and hover_at still snap a point to the listed control under it, but only when the control's own center is reachable. When the innermost node at the center belongs to another node (an overlay, or an enabled control nested inside it, like a card link around its "Add to cart" button), the runner sends the bare point through the engine's performAt instead (packages/e2e/src/agent/point-tap.ts, centerReaches).
  • An engine without a bare-point tap keeps tapping by id, as before.
  • The step summary says which happened: Tapped the point (187, 150) inside #n738 button "dismiss popup" instead of Tapped #n738 …, the control at (187, 150).
  • "Innermost" is the tree's stand-in for what is on top, the same rule hitTest already uses, not paint order. An overlay listed shallower than the control is not seen.
  • type_at, press_at, and select_at are unchanged: they act on a field, so the field's center is the right place.
  • Replay stays as durable as before: the recorder stores a bare point as within the node under it.

Verified

Ran it locally: yes

  • e2e mcp against Apple's UIKitCatalog on an iPhone SE (3rd gen) simulator, iOS 26.5: open the Date Picker popover, then tap_at at (187, 150), above it.
mainthis branch
Tapped #n738 button "dismiss popup" testid="PopoverDismissRegion", the control at (187, 150). The popover stays open.Tapped the point (187, 150) inside #n738 button "dismiss popup" testid="PopoverDismissRegion". The popover closes.
mainbranch
  • Mobile benchmark on two iPhone 17 Pro simulators (iOS 26.5), branch build: deterministic suite 55 passed, 1 skipped; agentic suite with a model key 21 of 22. The failure, Apple Pay Billing Address, is the simulator timing out while typing into the Apple Pay sheet, which uses no point tool.
  • The new unit and integration tests fail with the nested-control check or the engine-capability check removed.

Changes made by Claude Opus 5.5 in Claude Code.