#34637 test: evaluate prompt guidance on paired repository tasks

Leo-ask: ask-c344b7792d

Leo asked for agents to read things earlier and concurrently while multitasking, and to investigate whether the prompting could be improved. This adds a fixed paired prompt-behavior evaluator, plus call/turn and parallel metrics in the existing tool evaluator.

The standalone evaluator uses a captured Claude Code system prompt with its native Read/Bash schemas and deterministic repository fixtures. Baseline and candidate use the same three tasks and sample count with alternating arm order. It records prompt/fixture hashes, full transcripts, success, wall time, first-read latency, tokens, calls per tool turn, parallel share and background starts. The candidate is the compact asking rule plus the canonical early/concurrent-work guidance. No generated shell commands execute. This is synthetic prompt-behavior coverage, not DEV app or Codex verification.

Validation: Bun bundles all three changed modules; git diff --check and cmux-next prepush gate pass.

Paired exact-head evaluator receipt (local lane host, gateway-reachable): commit fd751732688cba73b423d31c6db7fe81c5350b88; ANTHROPIC_BASE_URL=http://sr2-gateway.tail137216.ts.net:4016 ANTHROPIC_AUTH_TOKEN=subrouter node tests/tool-evals/prompt-behavior.mjs --samples 3; baseline 9/9, guidance 9/9. Baseline medians: wall 7564 ms, first read 2734 ms, tool calls 4, tool turns 2, parallel share 0.5, tokens 13689. Guidance medians: wall 3247 ms, first read 1839 ms, tool calls 3, tool turns 1, parallel share 1, tokens 9084. The paired run is recorded at /home/cmux/lane-scratch/next-system-prompt-2/eval-local/log.

The fleet fast-check run 41794283465 could not reach sr2-gateway because its Linux workers are outside the tailnet (fetch failed); this is an infrastructure route limitation, not an evaluator or code failure. The same exact-head evaluator completed locally through the lane host's gateway route.

Metric	Before	After	Delta	Harness
Always-loaded asking/guidance text, estimated tokens	198	174	-24 (-12%)	Source chars / 4; paired manifest records exact text and method

Risk: evaluator-only. Remove prompt-behavior.mjs or omit --prompt-arm to revert the added evaluation path; application behavior is unchanged.

🤖 Generated with Codex

Agent session: agent session

---

^{Open in [code]smith · Need help on this PR? Tag @codesmith-bot with what you need.}

---

Summary by cubic

Evaluates whether the compact asking rule plus early/concurrent-work guidance changes model behavior on paired repository tasks, using A/B prompt-guidance instrumentation.

• The new standalone prompt-behavior.mjs runs three fixed tasks against deterministic fixtures with a captured Claude Code system prompt, alternating arm order per sample. It appends the guidance block to the first user turn to match product delivery and routes through the gateway via ANTHROPIC_BASE_URL and ANTHROPIC_AUTH_TOKEN. It records prompt hashes, transcripts, success, wall time, first-read latency, tokens, tool-turn counts, parallel share, and background starts. The baseline asking rule is frozen at a recorded source revision, and a guard rejects templates that already contain cmux guidance.
• The tool evaluator gains --prompt-arm none|baseline|guidance and the same tool-turn, parallel-share, and first-read metrics, delivering the candidate through the Claude preset's append field or the Codex developer-instruction field. The paired-run workflow is documented in the tool-evals README.
• Changes are evaluator-only: application behavior is unchanged, and the new path is reverted by deleting prompt-behavior.mjs or omitting --prompt-arm.