feat(mcp)!: take UI targets as top-level ref, selector, and x/y

Merged
TH
5d ago
+647-510

Summary

An agent without the schema loaded guesses the CLI's vocabulary, such as press {"ref":"@e6"}. The interaction tools wanted {"target":{"kind":"ref","ref":"@e6"}}, so every guess cost an INVALID_ARGS turn.

  • The shared command input in command-surface is now flat. press, click, fill, longpress, and hover take ref (with optional label), selector, or x+y. get takes ref or selector. kind is gone: the key that is present decides the form.
  • The CLI readers spread the positional parser's target into that same object. The MCP schema stays derived from the field metadata.
  • A nested target now fails with the top-level spelling.
  • Breaking: batch steps carry the same input. Saved batch --steps files and client.batch.run steps that use the nested target now fail with that migration message. The typed client.interactions.* API was already flat and is unchanged. client-api.md already said step input uses the client command's fields.
  • An unknown argument now answers with a call example the tool accepts, for example Hint: Example call: press {"ref":"@e12"}. Inside a batch, it answers with an example step.
  • 39 files.
press {"ref":"@e6"}        click {"selector":"id=\"home/products\""}        press {"x":201,"y":160}

Validation

  • validation: CI (head 39190fb4a1). Local runs were limited to focused vitest: command-surface, src/mcp, cli-grammar, src/commands, and the client index tests, all green.
  • Guard: in command-tools-parity.test.ts, CLI argv and MCP input reach runCommand as the identical object, for 6 commands across ref, ref with label, selector, point, and pinned ref.
  • Contract: every tool's hint example is accepted on retry.
  • Headless claude -p with only the local agent-device mcp, iPhone 17 Pro Max simulator, Settings app. press {"ref":"@e6"} and click {"selector":"label=\"Accessibility\""} both landed. 0 INVALID_ARGS, both with schemas loaded and with ToolSearch blocked.
  • CI owns unit-ci and the device lanes.

🤖 Generated with Claude Code


Autofix with [code]smith
Open in [code]smith · Need help on this PR? Tag @codesmith-bot with what you need.