refactor(evals): generalize the bench harness seam (registry-derived harnesses, shared resolvers, external-runner skeleton)

Merged
MI
2mo ago
+3269-841

What

Generalizes the packages/evals harness seam so the Nth external harness is a registry entry plus a trajectory adapter, instead of ~900 lines of copy-paste. Behaviour-preserving for claude_code/codex.

  • Registry-derived harness type — Harness is no longer a closed literal union; parseBenchHarness/isExecutableBenchHarness, help text and planner error copy derive from harnessRegistry. ClaudeCodeHarnessConfig/CodexHarnessConfig collapse into one ExternalHarnessConfig.
  • defineExternalHarness({ harness, supportedToolSurfaces, defaultModels, prepareToolAdapter, runAgent }) + registerBenchHarness — owns verifier-carrier construction, adapter prepare/cleanup and the generic start() error. claudeCodeHarness/codexHarness are expressed with it; default model lists move onto the definition (EVAL_<HARNESS>_MODELS override preserved).
  • Shared resolveToolSurface / resolveStartupProfile (framework/harnesses/toolSurfaceResolution.ts) replace the byte-identical per-harness resolvers.
  • runExternalHarnessTask skeleton (framework/harnesses/externalRunner.ts) — prompt assembly, EVAL_RESULT parsing (marker + structured-output contracts), bounded evidence capture, step-observation drain, grading, TaskResult assembly. Both runners use it.
  • Normalized harnessStatus/harnessStopReason and harness_input_tokens/harness_output_tokens/harness_total_tokens/harness_cost_usd metrics, emitted alongside the existing prefixed names; leaderboard alias table updated.
  • stagehand_facade gets captureEvidence — the runner now owns the facade stdio server and hands the agent a loopback relay, so facade runs produce a final url/ariaTree/screenshot like stagehand_code.
  • evals core --tool stagehand_facade is rejected up front instead of failing per case.

Testing

  • turbo build typecheck + full unit gate: evals 455/455 (was 418), all integrations suites green.
  • Live: exercised end-to-end by the mastra/pi/eve/deepagents harness smokes stacked on this PR (facade relay + evidence capture working on Browserbase).
  • Not yet re-smoked live with claude_code/codex on the facade surface (unit-tested against a fake JSON-RPC facade).

Summary by cubic

Refactors the eval harness layer so a new external harness is a registry entry plus a trajectory adapter instead of ~900 lines of copy-paste. claude_code and codex behavior is preserved, with normalized metrics (Codex no longer double-counts cached input or reasoning output) and up-front rejection of stagehand_facade for core runs.

Review/rollout

  • Focus: framework/benchHarness.ts (registry + defineExternalHarness/registerBenchHarness), framework/harnesses/externalRunner.ts, core/tools/stagehandFacadeBridge.ts, framework/harnesses/toolSurfaceResolution.ts, planner/runner/TUI.
  • Harness parsing, help text, and default-model (EVAL_<HARNESS>_MODELS) guidance all derive from the registry via listBenchHarnesses* and formatBenchHarnessFlags(); BenchHarness.start is optional, so planning-only harnesses are rejected in the runner and TUI.
  • Shared resolveToolSurface/resolveStartupProfile replace per-harness resolvers, and core dry-runs use listCoreRunnableTools(). Do not use --tool stagehand_facade with core runs; pick a harness (--harness claude_code or --harness codex) and a supported surface.
  • The external runner owns prompt assembly (marker or structured output), result parsing, bounded evidence capture, step-observation drain, grading, and TaskResult assembly; claude_code/codex runners are thin wrappers.
  • The runner spawns the facade MCP stdio server and exposes a loopback relay so facade runs capture final url/ariaTree/screenshot from the agent's browser; the bridge is split into FacadeProcess/RunnerRpcClient/AgentRelayServer with no behavior change.
  • The bridge hardens handshake replay/backpressure/fail-fast, cleans up carriers, and gates sdk_error with sanitized errors.
  • Normalized harness_* metrics emit alongside the legacy per-harness names, and TaskResult gains harnessStatus/harnessStopReason; unified parsing accepts marker and no-marker shapes, and SDK iteration errors take precedence over result text.