cmux NEXT MCP: tool evals on real Claude/Codex harnesses, and code mode takes the script shapes models write

Merged
LA
7d ago
+2486-58

Summary

Rounds 1 and 2 of the cmux MCP tool-eval hill climb. This PR now replaces #26625 and holds all of its commits, merged with current feat-cmux-next. Round 1 is in #26625's description; round 2 adds two things.

1. Evals that run the real agent harnesses (tests/tool-evals/)

  • Claude Code runs through the Claude Agent SDK. Codex runs through codex app-server. Each agent works in a clean home, with cmux MCP behind a logging relay (claude-cli:<model>, codex-cli:<model>). Any OpenRouter model runs in a generic, OpenCode-like harness.
  • --remote-daemon host:/path/cmux runs each sample's cmux daemon and MCP server on another host over ssh. The sample's terminals then use that host's PTYs. The person's Mac caps PTYs at 511, and their own cmux holds most of them.
  • The agents run sandboxed under a794's live guard (#05f7e8c). They cannot open the person's browser or cmux app. The sandbox also hides this machine's skills (an old installed cmux skill sent Codex to classic commands) and other samples' results (Codex searched those for answers).
  • sign-in-link-in-cmux moves to the app tier: headless has no cmux browser.

2. cmux MCP code-mode fixes, each from errors seen in the panels

  • A change's new ids are now also at the top of its answer (r.terminal_id as well as r.value.terminal_id). Six of nine models read the top level.
  • A terminal tool that names a tab gets that tab's terminal, in the tab's session and scope. This was 39 "needs terminal" errors across models. terminal_list keeps tab as a filter.
  • Tool calls return Promises, so .catch() works.
  • A script written as one anonymous function (async function () {}, async () => {}, (async () => {})()) runs and returns its value. Detection uses only the text; no script code runs to decide.
  • A missing required selector now names its fix, and setTimeout points to terminal_wait.
  • The tool sheet now runs commands in a shell tab that stays open, not in a shell tab that closes with its command.

Changelog

Changed: cmux MCP code mode accepts the script shapes models write (.catch, a function wrapper, a tab where a terminal is needed, ids at the top of a result), so agents waste fewer calls on cmux tasks.

Results

On the OpenRouter panel (9 models, 34 headless cases, 2 samples each), cases passed went from the #26625 build to this branch (h4):

Model#26625this branch
GLM 5.3 FlashX2029
Kimi K3.1 Pro2833
Grok 4.72831
DeepSeek v4.1-flash2931
Mistral Large 42731
Gemini 4 Argon3434
Qwen 3.8 Max3130
GLM 5.3 Prime2627
MiniMax M3.1 Flash2725

Tool errors fell for most models, for example GLM 5.3 FlashX from 59 to 28. Two models went down: Qwen 3.8 Max by 1 and MiniMax M3.1 Flash by 2.

The Claude SDK panel and the GPT 6 Astra panel (through codex app-server) are running again on 307b1fa. Two harness bugs invalidated the earlier runs: the remote MCP settings file was not written in time for 15 samples, and Codex was stuck at start under the guard. Both are fixed in this branch. The results will be posted as a comment.

Testing

  • cargo test --locked -p cmux-tui --lib -- cli::mcp: 123 passed, run on the rbx builder at 307b1fa.
  • cargo test --locked -p cmux-tui --test cli -- caller_selector: 4 passed.
  • cargo clippy --locked -p cmux-tui --lib --tests -- -D warnings: 0 warnings.
  • bun test tests/tool-evals/live-guard.test.mjs: 10/10.
  • The commits after 307b1fa change only tests/tool-evals.

Observed / Not run

  • Observed: all of the above. Before the fixes, mcp_current_is_the_callers_pane_without_an_app failed on 93729d3 (the earlier tab shortcut) and passed after it.
  • Not run: --surface all and cli after these changes; the app tier; Swift and app builds (the change is Rust only).

Reviews

Four correctness-review rounds by subagents.

  • Round 1: terminal_of_tab also rewrote terminal_list and dropped a tab's session. Fixed in ee8cf2b.
  • Round 2: the function-shape probe evaluated script text in an unlimited context, arrow forms were no longer wrapped, and a test could not fail. Fixed in f9b3d5f with a text-only check.
  • Round 3: SHIP. Its non-blocking notes (self-calling functions; a test case that could not fail) are fixed in eedd56a and 307b1fa.

Size: 1,711 of the 2,527 lines are self-contained eval tooling (tests/tool-evals, with a small change to tests/agent-evals snapshot.mjs) and contain no product code. The product change is 816 lines in cmux-tui/src/cli/mcp* with their tests. It carries all of #26625 (closed in favor of this PR), and the eval is the evidence for every product change.