fix(mobile): hide nodes under the soft keyboard and fail covered taps as actions
On a device, "visible" meant only "not hidden", and hidden came only from agent-device's visibleToUser: false. A row under the soft keyboard therefore counted as visible. scrollUntilVisible stopped on it, toBeVisible() passed, and the tap then failed as an engine error.
packages/mobile/src/nodes.ts:projectSnapshottakes akeyboardband. A node whose centre is in the band and none of whose children shows getsstates.hidden, the same as a node scrolled off screen.- The band comes from
keyboardBandinnodes.ts. It is agent-device's measured frame when there is one. Otherwise it spans from the top of the tree's keyboard elements (Keyboard, itsKeys, and theSystemInputAssistantViewsuggestion strip) to the bottom of the screen. On iOS 27, agent-device 0.21.36's simulator capture publishes no frame, so the fallback is the path that runs there. - Never hidden: the keyboard's own nodes (keys,
keyboardOwned, the window that holds the keyboard, later siblings that lie inside the band such as the globe and dictation dock), and the focused field. On a live device, iOS reported the focused search field at its frame from before the keyboard came up (y=789, drawn at about 511). packages/mobile/src/errors.ts: atarget_coveredortap_keyboard_occludes_targetrefusal withdispatched: 'no'now maps toNOT_ACTIONABLE(ACTION_FAILED), keeping agent-device's hint. Before, it wasENGINE_FAILURE, which counts as an infrastructure failure.- The golden test projects with the same band as
observe. Inios/sequential-onboarding.tree.txt, the Continue button under the keyboard is now[hidden]. The benchmark's onboarding tests dismiss the keyboard before they tap it. - Docs: a new Visibility section in
docs/reference/mobile.mdx, a note onscrollUntilVisibleinscreen.mdx, and theskills/e2esetup reference.
Not covered: an app view drawn over a node that agent-device's cover scan does not classify, such as the plain View footer in the issue. That node is still not hidden, and agent-device does not refuse a tap on it. I did not turn interactionBlocked: 'covered' into hidden, because a false positive there (ENG-968) would remove a real node from role queries.
Judgment call for review: a node listed after the keyboard under the same parent that starts inside the band counts as keyboard (the dock). A bare app control in that exact spot would stay visible. No capture shows that shape, and hiding the dock instead would drop real keyboard buttons (thread).
Issue
Part of ENG-1042
Verified
Ran it locally: yes
- Built CLI, iPhone 17 simulator (iOS 27, Korean locale), Contacts app. The test opens a new contact, taps the 성 field to raise the keyboard, then works with the 벨소리 row under the keyboard.
e2e run tests/scroll.e2e.ts:toBeHidden()on 벨소리,toBeVisible()on the focused 성 field, thenscrollUntilVisible(getByText('벨소리'))andtoBeVisible(). Passes on this branch.
| main | this branch | |
|---|---|---|
| 벨소리 row under the keyboard (y=783) before scrolling | visible | hidden |
scrollUntilVisible(getByText('벨소리')) | stops at once, row stays at y=783 under the keyboard | scrolls, row ends at y=494 |
getByText('벨소리').tap() with the keyboard up | ENGINE_FAILURE, exit 3 | ACTION_FAILED, exit 1 |
Branch output for the tap:
× |iphone| tests/keyboard.e2e.ts > tap under the keyboard 9.98s
ACTION_FAILED: perform tap failed: Ref @e72 is behind the visible keyboard and cannot be tapped safely Hint: The visible keyboard covers this target, so the tap would land on a key instead of on it. End editing first: ...
🤖 Generated with Claude Code