#58 Add eval harness + cases for all 10 agents

Repeatable dry-runs of the showcase agents, ported from the watchdog-agents eval harness. Each case fires one event at one agent's real handler and asserts routing + side effects. Complements the existing npm test unit suite (which uses hand-rolled ctx spies) by running the real handler through the runtime's simulation API and, in live mode, against an actual cheap model.

What's here

• scripts/evals/run-evals.mjs — two executors:
  • simulate (default): free, offline. Runs each handler through the runtime's simulateInvocation against an in-memory VFS; harness.run/llm.complete are stubbed but recorded as side effects. Asserts status, eventSource, sideEffectsAll/Any, logsAny.
  • live (--live): backs the model calls with a cheap opencode model (gpt-5-nano); --judge grades chat replies against the case rubric (LLM-as-judge).
  • Flat/nested agent layout; seeds materialized to both ctx.files and the disk mount so relayClient reads (linearClient().getIssue) resolve; new expect.status:"failed" + expect.errorIncludes for required-input guard cases.
• evals/cases.jsonl — 11 cases covering all 10 agents (linear-slack, linear, review, repo-hygiene, hn-monitor, spotify-releases, vendor-monitor, granola, both cloud-team members). 11/11 green in simulate.
• evals/seeds/*, evals/README.md — fixtures + docs.
• package.json: evals / evals:live scripts (compile personas first); tsx devDep. .gitignore: .evals/ + provider draft-tree safety net.

How to run

npm run evals                 # simulate, all cases
npm run evals -- --list
npm run evals:live -- --judge # real cheap-model replies + judge

Verification

tsc --noEmit clean · all 10 personas compile · npm run evals → 11/11.

🤖 Generated with Claude Code

---

Summary by cubic

Adds a repeatable eval harness that runs real agent handlers in simulate or live mode, with 11 cases across all 10 showcase agents. Improves coverage for routing, side effects, and chat reply quality.

• New Features

  • scripts/evals/run-evals.mjs: simulate (offline, stubbed harness.run/llm.complete) and live (--live via opencode, optional --judge); supports flat and nested agent dirs.
  • Seeds written to in-memory VFS and disk mount so client reads resolve; supports short and exact VFS-path seeds.
  • 11 cases in evals/cases.jsonl with fixtures and expectations, including required-input guard checks (expect.status: "failed" + errorIncludes).
  • Artifacts saved to .evals/runs/<stamp>/{result.json,summary.md}; evals/README.md documents usage and seeding; .gitignore ignores .evals/ and adds a provider draft-tree safety net.
  • package.json: evals / evals:live scripts (compile personas first) and tsx dev dependency.

• Bug Fixes

  • Updated tests/linear-slack-agent.test.mjs to map SLACK_CHANNEL to TEST_SLACK_CHANNEL via inputSpecs, preventing env collisions and aligning with the harness env mapping.