PR #26625 - manaflow-ai/cmux

Summary

This PR adds a tool-use eval for cmux's MCP and CLI (tests/tool-evals/), in the shape of t3code's evals/mcp-tool-use. It then hill-climbs the code-mode tool text against that eval on Sonnet 5.5 and Opus 5.5.

The eval. Each case replays one real Claude Code turn: a trimmed capture of the system prompt and native tools. The cmux text comes from the binary under test. The cmux side runs for real: an ephemeral headless daemon, cmux mcp serve, and the CLI. A stand-in app socket answers only system.identify, snapshot.get and a captured action catalogue. The cases (27 plus 8 agent-evals tasks) come from real failures that a transcript sweep found: 2,563 real cmux calls with an 8.3% error rate (details in tests/tool-evals/RESULTS.md).

Round 1 results (code mode, 2 samples per case):

Model	Cases pass	Cost	Input tokens	`cmux.docs` calls
Sonnet 5.5 before → after	28 → 33 of 35	$7.26 → $2.34	5.84 M → 2.86 M	122 → 18
Opus 5.5 before → after	30 → 34 of 35	$16.96 → $5.61	6.50 M → 3.22 M	137 → 36

No case that passed before fails after, on either model.

Product changes (cmux-tui/crates/cmux-tui/src/cli/mcp*):

cmux.docs matches a sentence by most of its words, not all of them, and drops stop words. It ranks the daemon tool before its app_* twin.
Docs answers are compact. They drop the protocol tag, the shared idempotency text and the per-selector paragraph, and they list the shared optional arguments in alsoTakes. Only cmux's own tools are compacted, and required arguments are kept.
The code-mode instructions now hold only the rules, and they fit Claude Code's 2048-character cut with the asking rule. The cmux_exec description now carries a common-tool sheet with argument names and result shapes (value.terminal_id).
An unknown tool called from a script now gets "did you mean ...".
With no app (cmux TUI or Cloud), current resolves from the daemon, as the CLI already does. A terminal with no place goes beside the chat, through the daemon's right neighbor. Links still go to the daemon's workspace.open_url.
Decimal fields (timeout_ms) take a whole JSON number. The schema type stays string.

Changelog

Changed: cmux MCP code mode finds tools from plain sentences, gives shorter docs answers and lists the common tools up front, so agents finish cmux tasks in fewer steps.

Testing

cargo test --locked -p cmux-tui --lib -- cli::mcp: 121 passed, on rbx.
cargo test --locked -p cmux-tui --test cli -- caller_selector: 4 passed, on rbx.
mcp_current_is_the_callers_pane_without_an_app was committed before its fix (4bc8f21f231). It failed on that commit with caller.unresolved, and it passes on the fix (7046be8c4ab).
cargo clippy --locked -p cmux-tui --lib --tests -D warnings, on rbx: no warnings.
Evals: bun tests/tool-evals/run.mjs --surface code --samples 2, on fleet builds of 7046be8c4ab (tool-evals-v1) and 065a20eb0df (tool-evals-v2). The final sheet text ran as overlays/v3.json over v2, and the same text is in the code.

Observed / Not run

Observed: the table above, plus the per-case results in RESULTS.md.
Not run: --surface all and --surface cli after the change. Codex models: the harness replays only Claude Code captures. The app tier (browser, Markdown). Swift and app builds: the change is Rust only.
open_url_without_an_app_stays_on_the_daemon guards the review's high finding. It runs without CMUX_TUI_TERMINAL_ID, so it also passes without the fix. The fix itself is the one-line Err(_) if terminal guard in placement.rs.

🤖 Generated with Claude Code

^{Need help on this PR? Tag @codesmith-bot with what you need.}

Summary by cubic

Adds a tool-use eval suite (tests/tool-evals/) that replays real agent turns against a real headless daemon and cmux mcp serve, then hill-climbs the code-mode tool text on Sonnet 5.5 and Opus 5.5. Both models pass more cases with roughly half the input tokens and about a third of the cost; no case that passed before fails after.

Eval harness

Each case replays one trimmed Claude Code turn; the cmux side runs for real, with a stand-in app socket answering only system.identify, snapshot.get, and the captured action catalogue.
A generic OpenCode-like prompt (cmux tools listed directly) and an OpenRouter adapter let any model run, not just Claude Code captures.
35 cases come from real failures a transcript sweep found across 2,563 real cmux calls (8.3% error rate), plus control tasks that must not touch cmux.
Results: Sonnet 28 → 33 of 35 cases, tokens 5.84M → 2.86M, cost $7.26 → $2.34; Opus 30 → 34 of 35, tokens 6.50M → 3.22M, cost $16.96 → $5.61.

Product changes

cmux.docs matches a sentence by most of its words, ignores stop words, ranks a tool's own name first and the daemon tool before its app_* twin; answers are compact, drop the protocol tag and shared idempotency text, and list shared optional arguments in alsoTakes.
Code-mode instructions fit Claude Code's 2048-character cut; the cmux_exec description carries a common-tool sheet with argument names and result shapes like value.terminal_id.
Unknown tools called from a script get "did you mean" suggestions.
With no app, current resolves from the daemon (as the CLI already does), and a terminal with no place opens beside the chat; links stay on the daemon's workspace.open_url.
Decimal fields like timeout_ms accept a whole JSON number; the schema type stays string.