PR #1559 - callstack/agent-device

Summary

#1542 describes two defects blocking the iOS checkout-form.ad replay corpus leg (and #1478 P5's evidence):

Scroll-inert: scroll down 0.6 reports success but the Form screen's ScrollView content doesn't move.
Recurring slow-AX deferral: a wedge where snapshot backends get deferred and dismiss-overlay fails.

This PR fixes defect 1's root cause and verifies the fix live. It does not make the corpus green — investigating defect 1 surfaced a second, distinct staleness bug in a shared cross-platform module that is now the sole remaining blocker. That one needs a design decision, so per the task brief I'm stopping short of unilaterally changing shared stabilization semantics and reporting it here instead.

Root cause (defect 1, fixed)

Dismissing the keyboard (keyboard dismiss, which taps the field's "Done" key) triggers the app's own ScrollView content-offset correction — releasing the inset it grew to keep the focused field above the keyboard. That correction is a separate, unsynchronized UIKit/RN animation that keyboard.waitForNonExistence (the only thing dismissKeyboard() waited on) knows nothing about: the keyboard's AX element can disappear well before the screen visually settles.

The very next command in the fixture is scroll down 0.6, which — like all iOS scroll/gesture commands — uses the AX-free synthesized drag lane (XCSynthesizedEventRecord/XCPointerEventPath, bypassing XCUIElement) specifically so scrolling keeps working when XCTest can't serialize the accessibility tree (the #1105/#758 AX-degradation lineage). That lane has no XCTest quiescence wait of its own. So the drag can land mid-animation, and its intended net displacement gets absorbed into stopping the residual content-offset correction instead of adding to it — the "scroll does nothing" symptom.

Evidence: xcrun simctl io recordVideo capture correlated frame-by-frame against the daemon's per-request timing (ios_runner_command_send spans with durationMs) shows the ScrollView jump to a wildly over-scrolled position during the keyboardDismiss window, well before the scroll command even executes — confirming the content is still animating when the drag lands. Reproduced deterministically (5/5 fresh-boot runs) via pnpm build && pnpm clean:daemon && node bin/agent-device.mjs replay examples/test-app/replays/checkout-form.ad --platform ios.

Fix

apple/runner/AgentDeviceRunner/AgentDeviceRunnerUITests/RunnerTests+Keyboard.swift: after the keyboard's AX element vanishes, dismissKeyboard() now polls screenshots (AX-free, same philosophy as the synthesized gesture lane) and only returns once 3 consecutive samples are byte-identical (or a 2s cap elapses). Requiring 3 matches — not 2 — is deliberate: a spring-driven correction can pass through a near-zero-velocity inflection that two adjacent samples can't distinguish from true rest.

The stopping decision (runnerScreenshotStabilitySettled) is extracted as a pure function and covered by unit tests under AGENT_DEVICE_RUNNER_UNIT_TESTS (5 cases: needs-enough-samples, matches-in-window, mid-window-mismatch, failed-capture never counts, only-looks-at-trailing-window). The capture/sleep loop around it is the thin, untestable I/O shell.

Scoped to iOS (#else branch of dismissKeyboard(), tvOS untouched); no cross-platform command contract or shared TS module touched.

Live validation (defect 1)

Check	Before fix	After fix
`scroll down 0.6` moves ScrollView content	No — content stays at pre-scroll offset (confirmed via video + fresh snapshot)	Yes — lands at the correct "Delivery choices" position (confirmed via video frame at the scroll's completion timestamp)
`gesture-lab.ad` (shared runner code, regression check)	—	2/2 clean fresh-boot runs, 31 steps each

Video captures and correlated frame extracts, request logs, and JSON failure payloads for every run (fresh-boot repros before the fix, and the fix verification) are preserved outside the worktree at /private/tmp/ad-1542-artifacts/ on the machine this was developed on (not included in the diff — this is evidence, not a repo asset).

Root cause (defect 2, NOT fixed — needs a decision)

With defect 1 fixed, checkout-form.ad still fails at the same step (click id="shipping-pickup" → "resolved to an off-screen element"), but the mechanism is now different and the on-screen content is actually correct at failure time (verified via video: "Pickup" is visibly on-screen at the moment of the reported failure).

src/daemon/post-gesture-stabilization.ts marks scroll/swipe actions for stabilization, then the next snapshot capture polls up to ~1.5s, comparing consecutive AX-derived "interaction surface signatures," and treats two matching consecutive reads as proof the screen settled. That assumption breaks specifically for the AX-free gesture lane: XCTest's AX tree isn't proactively resynced by a synthesized touch (unlike a normal XCUIElement.tap()), so it can serve a stale-but-internally-consistent tree for a window after the scroll — every poll matches the previous one, "stabilization" declares victory almost immediately (attempt 2, ~200ms in), and the click's off-screen guard then evaluates the pre-scroll node positions. Confirmed via the failure's own snapshotDiagnostics: captures are fast (p95 ~230ms, xctest backend, no slow/deferred flag) — this is stale-but-fast, not literally slow AX, which is why bumping timeouts wouldn't help; the loop exits on match, not on budget exhaustion.

This module is shared across iOS/Android (supportsPostGestureStabilization gates on isMobilePlatform, not a specific OS), so a correct fix changes cross-platform, cross-command stabilization semantics — e.g., cross-checking AX-signature stability against a screenshot signal, or treating "post-gesture signature identical to the pre-gesture baseline" as a "keep polling, don't trust this as final" signal rather than as proof of stability. Per the task brief, that's a design decision I'm not making unilaterally here.

Options for defect 2 (for discussion, not implemented)

Cross-check with a screenshot signal for scroll/swipe stabilization specifically — declare "stable" only when both the AX signature and a screenshot fingerprint agree across polls. Reuses the runnerScreenshotStabilitySettled pattern this PR introduces, but on the daemon/TS side and for a different trigger.
Baseline comparison: capture the interaction-surface signature before the gesture runs; if the post-gesture signature matches the pre-gesture baseline, don't trust "stable" — keep polling past the normal stabilization deadline (with its own bounded cap) or fall back to a screenshot check.
Force AX resync after an AX-free gesture via some XCTest-side invalidation, if such a hook exists, so the next snapshot query can't return the pre-gesture cached tree at all.

Any of these touches the shared stabilization module and needs sign-off before implementation.

Gates run

pnpm typecheck, pnpm lint, pnpm format:check — clean (no TS touched by this PR; run to confirm no incidental breakage)
pnpm build:xcuitest:ios — ** TEST BUILD SUCCEEDED **
gesture-lab.ad — 2/2 clean fresh-boot runs (regression check on the shared Swift file)
checkout-form.ad — still fails at step 11, now due to defect 2 (documented above), not defect 1

Test plan

Reviewer sign-off on the defect 2 approach (options above)
Once approach is picked: implement, then checkout-form.ad should pass 2/2 fresh-boot runs
Re-run pnpm test-app:replay:ios expecting both checkout-form.ad and gesture-lab.ad green