fix(open): wait for the launched app's first screen before the first capture

TH
3m ago
+741-101

Summary

On iOS, the first capture after open launches an app now waits for the app's first screen. This includes the open --foreground initial snapshot. Before this change, a React Native launch tree that held only the Application root settled on its first quiet pair, so open --relaunch --foreground returned a 1-node snapshot.

Rule (loop in post-gesture-stability.ts, predicate showsFirstScreen in deferred-interaction-outcome.ts, armed by markPostOpenStabilization on Apple only):

  • A tree qualifies when an element other than the Application/Window roots has a label or value that is not a busy indicator's, and two reads in a row agree (positions included, so a splash-to-app animation keeps polling).
  • A splash image, an empty view, or only a busy indicator never qualifies, however long it stays unchanged.
  • No node threshold or quiet window. An earlier revision accepted a tree with ≤4 nodes after 1.5 s unchanged. Bluesky traces showed why that fails: its storyboard splash is one unlabeled full-screen Image, and it held unchanged for 1.2–1.9 s after open returned. The empty React Native root that follows is one unlabeled view.
  • Trade-off: an app whose first screen has no labeled element (a game canvas) waits the full budget. An opt-out (open with settle: false) is a follow-up.
  • Polling budget 10s (a capture started in time still completes and is judged). When it runs out, the snapshot returns what it read with postGestureOutcome: {kind: "loading"} and a warning. is/wait absent reports observation: "loading".

Help and docs: the open --foreground CLI usage line and website/docs/docs/commands.md describe the wait. The always-loaded MCP open schema keeps its main wording, because it is within bytes of the 22 KB budget. An agent learns of a timed-out wait from the warning in the response.

observable opens arm the wait too. The open record replaces any stabilization or moving-target baseline an earlier action still owed: the open replaced that screen, so a verdict against it would charge the open to the earlier action. Android stays unchanged: its trees have no root node, so the content rule needs its own design.

Validation

Live run on head 72afa34 (readiness rule from 896ab43, budget fix 72afa34). iPhone 17 Pro sim (iOS 27.0), Agent Device Tester Debug dev build, Metro (expo start --dev-client) serving on 8081. The run used this branch's CLI (pnpm build, bin/agent-device.mjs) with its own --state-dir. Command: open com.callstack.agentdevicelab --relaunch --foreground --metro-port 8081 --json. Host load average was 90–128 during the runs.

RunTimeNodes (interactive)postGestureOutcomeBanner or root-only tree
first connect (cold Metro bundle, 7.4 s)17.0 s52noneno
relaunch 1–66.4, 6.9, 10.9, 7.0, 6.4, 6.4 s52 eachnoneno
warm open, app already running (n=3)2.2–2.3 s55noneno

All 7 relaunches returned the full home tree. The home screen has grown since the earlier run, so it is now 52 interactive nodes, not about 30. No run returned a Metro loading banner or a root-only tree. The first open, before Metro connected, returned the dev launcher (Development Build, DEVELOPMENT SERVERS). That is the dev client's own screen, not a loading surface.

Read-by-read trace of the same relaunch with the pre-PR CLI (0.21.36): open returned while the tree was root-only. That phase lasted about 1.3 s, and then the tree went 41 → 88 nodes (still changing) → the same 88 nodes twice. No read held a banner node. The Metro banner does not appear in the app's accessibility tree, so showsFirstScreen needs no banner rule.

Readiness rule change (896ab43): Bluesky Release build, iOS 27.0 sim, 4 launches traced read by read. The new predicate rejected every splash and empty-root read and accepted only the home screen. Tester Release, FlappyMorse (a game with labeled elements), and native Settings, Contacts, Calendar, Maps, Photos, Files, Reminders, Health and Safari show labeled trees by the time open returns.

Earlier runs (before 896ab43, under the replaced sparse-tree rule): base 80075b6 --relaunch returned 1 node 10/10 (about 2.7 s). The previous rule returned 30 nodes 0/22 sparse (3.5–9.5 s, median about 4.5 s). Warm open went from a 625 ms median to 1004 ms (n=15). These no longer describe the current iOS logic. The live run above replaces them.

CI owns the full suite. Locally, with main merged: typecheck, format, the MCP budget test, and the touched capture-kit, daemon, command-surface and MCP suites passed.

Risks: providers that report no postOpenObservation do not arm the wait. An app whose tree stays root-only (empty UI or accessibility unavailable) waits the full 10s and reports loading.

🤖 Generated with Claude Code


Autofix with [code]smith
Open in [code]smith · Need help on this PR? Tag @codesmith-bot with what you need.