fix(mobile): read an empty iOS text field as empty, not as its placeholder

Merged
KA
7d ago
+31-7

On iOS, an empty text field read as if it held its placeholder text. That broke the ios expo agent test in the nightly examples run on both attempts.

  • agent-device reports an empty React Native TextInput on iOS with value equal to placeholder and no hintShowing flag. Only UIAutomator sets that flag; the iOS runner has no such field:

    {"type":"TextField","label":"Name","value":"Ada Lovelace","placeholder":"Ada Lovelace","identifier":"name","editable":true}

  • packages/mobile/src/nodes.ts blanked the value only on hintShowing, so the field became textbox "Name" value="Ada Lovelace". The agent's view hides placeholder when a field has a name (packages/e2e/src/agent/observation.ts:448), so the model saw only a field it thought was already filled.

  • Asked to "press Greet without typing a name", the agent spent its 11 turns trying to delete "Ada Lovelace" and never pressed Greet. The run before it read the field the same way and passed by luck.

  • The reader now treats an editable iOS field whose value equals its placeholder (trimmed) as empty. agent-device's runner uses the same check during text entry (textMatchesPlaceholder). Android still relies on hintShowing.

  • History: #499 hit this on the CI Mac and added the hintShowing check, believing the iOS bridge set the flag. No agent-device version does that on iOS: 0.21.13, the version #499 used, and 0.21.28 both read it from Android sources only. The test that check came with fed iOS a flag that never comes. agent-device later added placeholder to iOS nodes so consumers could tell an empty field from a filled one (callstack/agent-device#2961). The reader never used it for that until now.

  • Trade-off: on iOS, text typed to exactly match the placeholder also reads as empty. The snapshot cannot tell the two apart. docs/reference/mobile.mdx now says so.

  • The iOS goldens in packages/mobile/tests/fixtures/snapshots/ios/ do not change. In those captures of the mobile benchmark, empty fields come with no value at all. The unit test pins the shape captured from examples/with-expo.

Verified

Ran it locally: yes

  • examples/with-expo built for the iOS simulator as the nightly builds it (expo prebuild, Release xcodebuild), on an iPhone 17 Pro with iOS 26.5, Xcode 27.1, and agent-device 0.21.28, the same agent-device version the nightly used.
  • A probe test dumped the screen the agent reads in three states. main used the npm packages; this branch used e2e, @e2e-dev/web, and @e2e-dev/mobile packed from the branch, the way the nightly swaps them in:
statemainthis branch
empty, after app.open()textbox "Name" testid="name" value="Ada Lovelace"textbox "Name" testid="name"
empty, focusedtextbox "Name" testid="name" value="Ada Lovelace" [focused]textbox "Name" testid="name" [focused]
after fill('Ada')value="Ada"value="Ada"
after Greet clears ittextbox "Name" testid="name" value="Ada Lovelace"textbox "Name" testid="name"
  • The example's tests/deterministic.e2e.ts passes on the branch build.
  • Not run: the example's agent test. No model key was set locally.