feat: add selector-targeted drag gestures

Merged
TH
3mo ago
+2744-178

Summary

Adds a generic selector/ref-targeted drag gesture across the CLI, Node client, MCP, recording, and replay surfaces.

  • Resolves source and destination independently before dispatch, then lowers their centers to one uninterrupted pointer plan: source hold → timed movement → optional destination hold → release.
  • Accepts selectors or pinned snapshot refs at either endpoint and returns selector chains plus resolution disclosures for both.
  • Records portable selector chains and one targets-v1 annotation containing source and destination identity evidence; replay verifies both endpoints before pointer-down and attributes a later guard mismatch to the correct endpoint.
  • Applies ref-admission and frame-expiry policy to both endpoints.
  • Supports target-authored drag only where the backend can preserve every authored phase: Android touch devices and iOS/iPadOS. Android TV, tvOS, macOS, visionOS, watchOS, Linux, Vega, and web reject it before injection.
  • Contains no react-native-reorderable-specific selectors, labels, fixtures, prototypes, or assets.

Public API

CLI:

agent-device gesture drag 'id="drag-source"' 'id="drop-target"'
agent-device gesture drag @e4~s12 'label="Archive"' 700 600 200

Node:

await client.interactions.drag({
  source: 'id="drag-source"',
  destination: 'id="drop-target"',
  sourceHoldMs: 700,
  moveMs: 600,
  destinationHoldMs: 200,
});

MCP:

{
  "kind": "drag",
  "source": "id=\"drag-source\"",
  "destination": "id=\"drop-target\"",
  "sourceHoldMs": 700,
  "moveMs": 600,
  "destinationHoldMs": 200
}

Defaults are 800 ms source hold, 500 ms movement, and 0 ms destination hold. The combined plan is capped at 10 seconds.

Recording and replay

A recorded drag stores one annotation of this shape:

# agent-device:targets-v1 {"source":{...},"destination":{...}}

Snapshot refs are materialized to portable selector chains. Strict publication refuses either endpoint when it remains a session-local ref or when dual identity evidence is missing. Replay resolves and verifies source and destination sequentially before dispatch, then threads both verified guards through the daemon boundary.

Validation

  • pnpm check:affected --run
    • 398 test files passed
    • 3,764 tests passed
    • 94.74% changed-line coverage (126/133)
    • 87.65% changed-branch coverage (142/162, non-gating)
    • format, lint, typecheck, layering, fallow, MCP metadata, build, package, provider integration, Node integration, and replay compatibility passed
  • Focused replay/publication regression set: 5 files, 62 tests passed.
  • Interaction-guarantee contract: 8 tests passed after merging the selector-package extraction from current main.
  • Android touch and iOS simulator fixture scenarios cover the supported device paths; capability tests explicitly cover every rejected platform listed above.
  • Exact-head public drag replays pass on a fresh Pixel 10 Pro emulator (8/8 steps) and iPhone 17 Pro simulator (7/7 steps), each guarded by drag completed yes.
  • Exact-head full-platform evidence: Replay Nightly run 30954741944 proves Android destination arrival in the full-emulator job and the named iOS destination step passes in the full iOS job.

Video evidence

https://github.com/user-attachments/assets/a075d604-09a6-4274-ac83-4f357745e5fb