fix: add iOS private AX snapshot fallback

Merged
TH
4mo ago
+1248-73

Summary

Adds a simulator-only private AX snapshot fallback for iOS when XCTest returns a sparse application/window tree or fails while serializing AX snapshots.

The fallback uses XCTest private accessibility interfaces dynamically, maps the recovered tree back into existing SnapshotNode output, and keeps normal XCTest snapshots authoritative whenever they return real content. It also adds a conservative public XCTest-query recovery tier for regular sparse snapshots before falling back to private AX; compact interactive snapshots still use the private-AX/find recovery path because Bluesky showed public queries collapse there too.

Also fixes SpringBoard permission alerts in compact interactive snapshots: modal detection now runs before the compact app-tree shortcut, and permission sheets return alert text plus actionable button refs without broad SpringBoard subtree walks.

Hardens mutating find actions on iOS: when compact interactive snapshots collapse to the application root, find retries with a full snapshot, then with a query-scoped full snapshot if unscoped AX serialization fails on unrelated feed content.

Closes #701
Closes #761

Validation

  • pnpm build
  • pnpm check:quick
  • pnpm check:unit outside sandbox after sandbox listener failures
  • pnpm build:xcuitest
  • pnpm exec vitest run src/daemon/handlers/tests/find.test.ts
  • Compared callstack/agent-device#761 locally on Bluesky Home: its public-query recovery did not fix snapshot -i -c, did not expose the permission alert, and find Search click failed there.
  • Prototyped #761-style public query recovery on this branch. Kept it only for regular sparse snapshots; compact public-query recovery and a direct KVC AX-object bridge were tested on Bluesky and dropped because they still returned only the application root.
  • Manual iOS simulator smoke: opened Settings on iPhone 17 iOS 26.2 and verified snapshot -i -c returned usable refs with the rebuilt runner.
  • Manual Bluesky iOS 27 simulator smoke with Xcode 26.2: opened ~/Developer/Bluesky, logged in, reached Home/Discover, verified full Home snapshot exposed 133 nodes including homeScreen, composeFAB, feed tabs, bottom-bar IDs, and post content.
  • Manual Bluesky iOS 27 find smoke after simulator reboot: compact Home snapshot remained sparse by design, but find Search click succeeded through the fallback and navigated to Explore/Search; find Search exists also passed.
  • Manual Bluesky iOS 27 permission smoke: reset app privacy, triggered the native Photos permission sheet from Home, verified snapshot -i -c returned an Alert node with title/message and Select Photos, Allow Full Access, and Don’t Allow button refs, then clicked the Don’t Allow ref successfully.

Touched files: 8. Scope covers the iOS XCTest runner snapshot path plus daemon find fallback handling for sparse compact iOS snapshots.