fix(mobile): read Compose Multiplatform text fields on iOS

Merged
KA
18d ago
+160-14

Fixes the iOS Kotlin Multiplatform job in last night's examples nightly (run 40592743298). Also fixes the iOS Kotlin Multiplatform and Flutter fill failures that #1410 brought to main.

What changed

  • Placeholder (packages/mobile/src/nodes.ts). On iOS, an empty Compose Multiplatform field reports no placeholder. It adds the text it draws to its label instead, as "Name, Ada Lovelace", over one StaticText child. The agent read that as a filled field and tried to clear it until the step timed out. Now, on an empty, editable iOS field, a trailing ", <text>" that matches the field's only text child becomes the placeholder attribute, and that child is left out. The agent sees textbox "Name" testid="name".
  • fill / clear guard (packages/mobile/src/surface.ts). #1410 refused every role except textbox, searchbox and combobox. agent-device 0.21.33's simulator AX tree reports Compose Multiplatform and Flutter fields as an unclassified Other. So the guard now also lets through an iOS other that holds a value or only text, and checks again against the snapshot taken just before typing. A childless, valueless other (a React Native Pressable) and a container of controls are still refused.
  • The mobile reference docs, both copies of the skill's setup.md, and a changeset are updated.

The placeholder logic needs the field to read as editable. agent-device's XCTest tree reports it that way already. Its simulator AX tree, which the nightly uses, will only do so after callstack/agent-device#4008. A follow-up PR bumps agent-device once that ships.

Verified

Ran it locally: yes

  • examples/with-kotlin-multiplatform on an iPhone 18 Pro simulator (iOS 27). The suite ran with packages packed from main and from this branch, both with agent-device 0.21.33, using e2e run --target ios --no-cache --retries 0 --video.
mainthis branch
mainbranch

main: 2 failed, 3 passed.

UNSUPPORTED_CAPABILITY: agent-device can only fill editable fields; node n70 is other
AUTOMATION_UNSUPPORTED: agent.act blocked: ... the agent could not replace the existing “Ada Lovelace” value with “Grace” ...

This branch: 5 passed, including both agent tests with live model calls. The same suite also passes 5 of 5 against the agent-device build from #4008.

Not run locally: @e2e-dev/testbed, because another checkout was holding its port, and the mobile benchmark. The mobile benchmark runs in the mobile workflow on this PR.

https://github.com/user-attachments/assets/b1fa982d-0cbc-40e4-88c0-254d8b04f717

https://github.com/user-attachments/assets/a9c200b3-fa08-4a86-8e94-3067da313c89