feat(mcp)!: take UI targets as top-level ref, selector, and x/y
Merged
TH
5d ago+647+647+012345678901234567890123456789-510-510-012345678901234567890123456789
Summary
An agent without the schema loaded guesses the CLI's vocabulary, such as press {"ref":"@e6"}. The interaction tools wanted {"target":{"kind":"ref","ref":"@e6"}}, so every guess cost an INVALID_ARGS turn.
- The shared command input in
command-surfaceis now flat.press,click,fill,longpress, andhovertakeref(with optionallabel),selector, orx+y.gettakesreforselector.kindis gone: the key that is present decides the form. - The CLI readers spread the positional parser's target into that same object. The MCP schema stays derived from the field metadata.
- A nested
targetnow fails with the top-level spelling. - Breaking: batch steps carry the same input. Saved
batch --stepsfiles andclient.batch.runsteps that use the nestedtargetnow fail with that migration message. The typedclient.interactions.*API was already flat and is unchanged.client-api.mdalready said step input uses the client command's fields. - An unknown argument now answers with a call example the tool accepts, for example
Hint: Example call: press {"ref":"@e12"}. Inside a batch, it answers with an example step. - 39 files.
press {"ref":"@e6"} click {"selector":"id=\"home/products\""} press {"x":201,"y":160}
Validation
- validation: CI (head 39190fb4a1). Local runs were limited to focused vitest: command-surface, src/mcp, cli-grammar, src/commands, and the client index tests, all green.
- Guard: in
command-tools-parity.test.ts, CLI argv and MCP input reachrunCommandas the identical object, for 6 commands across ref, ref with label, selector, point, and pinned ref. - Contract: every tool's hint example is accepted on retry.
- Headless
claude -pwith only the localagent-device mcp, iPhone 17 Pro Max simulator, Settings app.press {"ref":"@e6"}andclick {"selector":"label=\"Accessibility\""}both landed. 0INVALID_ARGS, both with schemas loaded and with ToolSearch blocked. - CI owns unit-ci and the device lanes.
🤖 Generated with Claude Code
![Autofix with [code]smith](https://pr-comments-assets.blacksmith.sh/codesmith/autofix-with-codesmith-dark-3x.png)
Open in [code]smith · Need help on this PR? Tag @codesmith-bot with what you need.