fix(mobile): read Compose Multiplatform text fields on iOS
Fixes the iOS Kotlin Multiplatform job in last night's examples nightly (run 40592743298). Also fixes the iOS Kotlin Multiplatform and Flutter fill failures that #1410 brought to main.
What changed
- Placeholder (
packages/mobile/src/nodes.ts). On iOS, an empty Compose Multiplatform field reports no placeholder. It adds the text it draws to its label instead, as"Name, Ada Lovelace", over oneStaticTextchild. The agent read that as a filled field and tried to clear it until the step timed out. Now, on an empty, editable iOS field, a trailing", <text>"that matches the field's only text child becomes theplaceholderattribute, and that child is left out. The agent seestextbox "Name" testid="name". fill/clearguard (packages/mobile/src/surface.ts). #1410 refused every role except textbox, searchbox and combobox. agent-device 0.21.33's simulator AX tree reports Compose Multiplatform and Flutter fields as an unclassifiedOther. So the guard now also lets through an iOSotherthat holds a value or only text, and checks again against the snapshot taken just before typing. A childless, valuelessother(a React NativePressable) and a container of controls are still refused.- The mobile reference docs, both copies of the skill's
setup.md, and a changeset are updated.
The placeholder logic needs the field to read as editable. agent-device's XCTest tree reports it that way already. Its simulator AX tree, which the nightly uses, will only do so after callstack/agent-device#4008. A follow-up PR bumps agent-device once that ships.
Verified
Ran it locally: yes
examples/with-kotlin-multiplatformon an iPhone 18 Pro simulator (iOS 27). The suite ran with packages packed frommainand from this branch, both with agent-device 0.21.33, usinge2e run --target ios --no-cache --retries 0 --video.
| main | this branch |
|---|---|
main: 2 failed, 3 passed.
UNSUPPORTED_CAPABILITY: agent-device can only fill editable fields; node n70 is other
AUTOMATION_UNSUPPORTED: agent.act blocked: ... the agent could not replace the existing “Ada Lovelace” value with “Grace” ...
This branch: 5 passed, including both agent tests with live model calls. The same suite also passes 5 of 5 against the agent-device build from #4008.
Not run locally: @e2e-dev/testbed, because another checkout was holding its port, and the mobile benchmark. The mobile benchmark runs in the mobile workflow on this PR.
https://github.com/user-attachments/assets/b1fa982d-0cbc-40e4-88c0-254d8b04f717
https://github.com/user-attachments/assets/a9c200b3-fa08-4a86-8e94-3067da313c89