PR #1563 - callstack/agent-device

Summary

#1542 defect 2 (documented in #1559's body, not implemented there pending sign-off): src/daemon/post-gesture-stabilization.ts marks scroll/swipe actions for stabilization, then treats two consecutive matching AX-signature polls as proof the screen settled. On iOS's AX-free synthesized gesture lane, XCTest's tree can serve a stale-but-internally-consistent read for a window after the gesture — every poll matches the previous one, "stabilization" declares victory almost immediately, and the next interaction evaluates pre-gesture node positions.

Mechanism

Baseline capture: markPostGestureStabilization now captures the interaction-surface signature from session.snapshot — the last-known pre-gesture capture, already available at the call site (this fires post-dispatch, pre-capture) — no extra capture added to the hot path.
Distrust decision: decidePostGestureStabilityVerdict (pure: {needsBaselineDistrust, baselineSignature, quietSignature, elapsedMs, distrustCapMs} → 'trust' | 'distrust' | 'accept-stale') fires only once a quiet poll-to-poll match is already observed. If that quiet signature still matches the pre-gesture baseline, it is not trusted — polling continues past the normal 1.5s deadline up to a bounded 3.5s cap (STABILIZATION_DEADLINE_MS + 2_000, real margin over both the 200ms poll interval and the normal deadline — see the repo's own zero-margin flake history before touching these numbers). On cap expiry with the signature still identical, the result is accepted (a genuine no-op gesture, e.g. scroll at an edge, is the honest answer) but flagged via a new post_gesture_snapshot_stale_accept diagnostic (matchedPreGestureBaseline: true) so a stale-accept is observable in ndjson.
Comparison is subset-tolerant, not whole-array equality (interactionSurfaceMatchesBaseline in src/daemon/interaction-outcome-policy.ts): the pre-gesture baseline and the post-gesture capture are routinely fetched with different snapshot scopes (e.g. a broad wait-driven capture vs. an interactive-only selector-resolution capture), so naive whole-array equality reports "changed" from scope drift alone. My first implementation used whole-array equality and it failed on the very first live run for exactly this reason (verdict was always trust on the first quiet match, never once catching the real staleness) — root-caused and fixed before shipping, see Adjudication below.

Platform scoping

Gated to Apple only via requiresPostGestureBaselineDistrust (isApplePlatform(device.platform)). Android's persistent helper clears its accessibility-node cache before every capture — android/snapshot-helper/src/main/java/com/callstack/agentdevice/snapshothelper/AccessibilityTreeCapture.java's capture() calls clearAccessibilityCache() (UiAutomation.clearCache() on API 34+, setServiceInfo(getServiceInfo()) fallback below it) unconditionally before every traversal, for both one-shot and persistent-session capture — so an Android post-gesture read is fresh by construction and never needs this check (#1254/#1259). Android never computes a baselineSignature at all (markPostGestureStabilization skips it), so this fix costs Android nothing — latency and semantics unchanged, live-confirmed below.

Counterfactual proof (docs/agents/testing.md)

Neutralized interactionSurfaceMatchesBaseline to return false unconditionally (reproducing "no comparable baseline evidence, ever") and reran the owning test files. 8 of 31 tests went red, including the async integration test that pins the exact live regression:

 × interactionSurfaceMatchesBaseline matches identical signatures
 × interactionSurfaceMatchesBaseline matches through a broader baseline scope when the shared element is frozen
 × interactionSurfaceMatchesBaseline matches through a broader current scope when the shared element is frozen
 × interactionSurfaceMatchesBaseline tolerates tiny rect drift on the shared element
 × decidePostGestureStabilityVerdict distrusts a quiet signature matching the baseline before the cap
 × decidePostGestureStabilityVerdict accepts a baseline-matching signature once the cap expires
 × capturePostGestureStabilizedResult keeps polling past the normal deadline when the AX tree is stuck at the pre-gesture baseline (iOS)
 × capturePostGestureStabilizedResult catches a frozen target even when the baseline came from a broader-scope capture than the post-gesture reads (iOS, live regression)

Test Files  2 failed (2)
     Tests  8 failed | 23 passed (31)

Restored the fix, reran — 31/31 green.

Live validation and adjudication

Ran on iPhone 17 (D74E0B66 — iPhone 17 Pro was lease-held; same-generation substitute, precedent from prior sessions), hardware keyboard off, pnpm build && pnpm clean:daemon, explicit --state-dir.

Check	Result
`gesture-lab.ad` (iOS, fresh-boot)	2/2 clean — no regression
`checkout-form-android.ad` + `gesture-lab-android.ad` (Pixel_7_CI, Release APK, freezer disabled)	2/2 clean — platform scoping confirmed safe
`checkout-form.ad` (iOS, fresh-boot)	Still fails at step 11 — expected residual, root-caused below

The fix demonstrably catches the literal defect described in #1559: one fresh-boot run of checkout-form.ad showed the entire post-gesture signature frozen at the exact pre-gesture baseline for the full distrust window, correctly flagged rather than silently trusted:

{"level":"warn","phase":"post_gesture_snapshot_stale_accept","command":"click","data":{"action":"scroll","attempts":45,"durationMs":21542,"matchedPreGestureBaseline":true}}

(That run used a temporarily-widened 22s cap purely to test whether more time ever resolves it — it didn't, which is why the shipped cap stays at the reasoned 3.5s: a bigger bound doesn't help this specific staleness, it only costs latency on every genuinely inert gesture.)

But checkout-form.ad still fails, for a different reason this fix correctly stops masking. A second fresh-boot repro resynced within 1.5s and differed from the pre-gesture baseline (post_gesture_snapshot_stabilized, attempts:3, durationMs:1566 — correctly trust, not distrusted) — yet the click still failed "off-screen". The raw error details show why:

"rect": {"x": 126.33, "y": 136.67, "width": 74.67, "height": 37.67},
"viewport": {"x": 18, "y": 381.0, "width": 366, "height": 109.33},
"scrollDirection": "up"

rect (the "Pickup" button) is correct — it matches the screenshot and the button's own post-scroll position. viewport — the nearest scrollable ancestor's own frame, read by findNearestScrollableAncestorRect/resolveEffectiveViewportRect in src/snapshot/mobile-snapshot-semantics.ts and treated as authoritative by the off-screen guard — is corrupted: the "Checkout form" ScrollView's frame reads (18, 381.0, 366, 109.33) post-gesture, when its true, pre-gesture frame (captured moments earlier via a clean snapshot -i) was (18, 62, 366, 729). A ScrollView's own frame does not change size when its content scrolls — this is neither the correct value nor the pre-gesture baseline value, so my distrust check (which only ever compares against the pre-gesture baseline) cannot catch it by construction: it's a third, wrong value. A same-selector retry immediately after (once postGestureStabilization clears and the daemon takes the direct-XCUIElement fast path in direct-ios-selector.ts, which bypasses the bulk snapshot walk entirely) resolves the tap instantly and correctly, confirming per-element direct reads are reliable and the corruption is specific to the bulk-snapshot-derived ScrollView ancestor frame.

Evidence preserved outside the worktree at /private/tmp/ad-defect2-artifacts/:

manual-probe/before-scroll.png, after-scroll-0ms.png — screenshots proving the gesture visibly works (refutes an inert-gesture explanation)
manual-probe/pre-scroll-tree.json, post-scroll-tree.json — full snapshot pairs showing the ScrollView ancestor's frame corruption alongside correct child positions
ios-diag/ — the 45-attempt/21.8s frozen-baseline capture (quoted above)
ios-run1/, ios-run2/ — checkout-form.ad fresh-boot failures with this fix applied
ios-gesture-lab-run1/, ios-gesture-lab-run2/ — gesture-lab.ad 2/2
android-run2/ — Android suite 2/2

Related: #1542 — deliberately not closed here. This fix is correct and safe as far as it goes (catches the literal frozen-tree case, zero Android cost, no regression on the passing scenario), but checkout-form.ad still does not pass, so #1542 stays open on its own merits. The residual blocker — the corrupted ScrollView-ancestor-frame read findNearestScrollableAncestorRect trusts — is a distinct defect from the one this PR addresses; a separate refusal-double-check fix (re-verifying an off-screen rejection through the direct-selector path before failing, or an equivalent) is being scoped separately.

Follow-ups

Mutation lane: neither src/daemon/post-gesture-stabilization.ts nor src/daemon/interaction-outcome-policy.ts is in scripts/mutation/modules.ts's KERNEL_MODULES, so pnpm mutation:run was not run against them per the task's own instruction not to add lane membership unilaterally. Worth considering given both now house pure agent-facing decision functions.
ScrollView-ancestor-frame corruption (new, tracked informally here pending its own issue): needs a design decision the same way this one did.

Gates run

pnpm typecheck && pnpm lint && pnpm format:check && pnpm check:layering && npx vitest run src/daemon/__tests__/post-gesture-stabilization.test.ts src/daemon/__tests__/interaction-outcome-policy.test.ts — all clean, chained in one run before pushing. Full npx vitest run (5363 tests) also run earlier in this branch's history: green except the pre-existing src/platforms/apple/core/__tests__/apps.test.ts contention-retry file (isolated rerun 54/54 green, unrelated to this diff).

Generated by Claude Code