fix(mobile): hide nodes under the soft keyboard and fail covered taps as actions

Merged
KD
4d ago
+362-27

On a device, "visible" meant only "not hidden", and hidden came only from agent-device's visibleToUser: false. A row under the soft keyboard therefore counted as visible. scrollUntilVisible stopped on it, toBeVisible() passed, and the tap then failed as an engine error.

  • packages/mobile/src/nodes.ts: projectSnapshot takes a keyboard band. A node whose centre is in the band and none of whose children shows gets states.hidden, the same as a node scrolled off screen.
  • The band comes from keyboardBand in nodes.ts. It is agent-device's measured frame when there is one. Otherwise it spans from the top of the tree's keyboard elements (Keyboard, its Keys, and the SystemInputAssistantView suggestion strip) to the bottom of the screen. On iOS 27, agent-device 0.21.36's simulator capture publishes no frame, so the fallback is the path that runs there.
  • Never hidden: the keyboard's own nodes (keys, keyboardOwned, the window that holds the keyboard, later siblings that lie inside the band such as the globe and dictation dock), and the focused field. On a live device, iOS reported the focused search field at its frame from before the keyboard came up (y=789, drawn at about 511).
  • packages/mobile/src/errors.ts: a target_covered or tap_keyboard_occludes_target refusal with dispatched: 'no' now maps to NOT_ACTIONABLE (ACTION_FAILED), keeping agent-device's hint. Before, it was ENGINE_FAILURE, which counts as an infrastructure failure.
  • The golden test projects with the same band as observe. In ios/sequential-onboarding.tree.txt, the Continue button under the keyboard is now [hidden]. The benchmark's onboarding tests dismiss the keyboard before they tap it.
  • Docs: a new Visibility section in docs/reference/mobile.mdx, a note on scrollUntilVisible in screen.mdx, and the skills/e2e setup reference.

Not covered: an app view drawn over a node that agent-device's cover scan does not classify, such as the plain View footer in the issue. That node is still not hidden, and agent-device does not refuse a tap on it. I did not turn interactionBlocked: 'covered' into hidden, because a false positive there (ENG-968) would remove a real node from role queries.

Judgment call for review: a node listed after the keyboard under the same parent that starts inside the band counts as keyboard (the dock). A bare app control in that exact spot would stay visible. No capture shows that shape, and hiding the dock instead would drop real keyboard buttons (thread).

Issue

Part of ENG-1042

Verified

Ran it locally: yes

  • Built CLI, iPhone 17 simulator (iOS 27, Korean locale), Contacts app. The test opens a new contact, taps the 성 field to raise the keyboard, then works with the 벨소리 row under the keyboard.
  • e2e run tests/scroll.e2e.ts: toBeHidden() on 벨소리, toBeVisible() on the focused 성 field, then scrollUntilVisible(getByText('벨소리')) and toBeVisible(). Passes on this branch.
mainthis branch
벨소리 row under the keyboard (y=783) before scrollingvisiblehidden
scrollUntilVisible(getByText('벨소리'))stops at once, row stays at y=783 under the keyboardscrolls, row ends at y=494
getByText('벨소리').tap() with the keyboard upENGINE_FAILURE, exit 3ACTION_FAILED, exit 1

Branch output for the tap:

× |iphone| tests/keyboard.e2e.ts > tap under the keyboard 9.98s
ACTION_FAILED: perform tap failed: Ref @e72 is behind the visible keyboard and cannot be tapped safely Hint: The visible keyboard covers this target, so the tap would land on a key instead of on it. End editing first: ...

🤖 Generated with Claude Code