feat(evals): add eve bench harness (integrations/eve-sdk)

Merged
MI
2mo ago
+3193-28

What

Adds --harness eve to evals, backed by a new private @browserbasehq/stagehand-integrations-eve-sdk package.

  • packages/integrations/eve-sdk — runEveSession: eve 0.29.4 has no in-process run-agent SDK and no stdio MCP support, so the session layer spawns eve dev --no-ui --port 0 per row against a generated agent app and consumes the HTTP NDJSON event stream (actions.requested / action.result / turn boundaries); tool-step budget counted on completed results; abort → session.cancel() + dev-server SIGTERM; redaction.
  • packages/evals/framework/eveToolAdapter.ts — generates the eve agent app (agent/tools/<server>__<tool>.ts proxies over an @modelcontextprotocol/sdk stdio client) for via:"mcp" mounts; bounded listTools/cleanup; mkdtemp rollback; load_skill and other non-browser framework tools disabled.
  • packages/evals/framework/harnesses/eveAdapter.ts — events → NormalizedToolCall[].
  • Registered via defineExternalHarness. Requires Node ≥ 24 (eve's own floor).

Testing

  • Full unit gate green (evals + eve-sdk 16 + all other suites).
  • Connected smoke (evals run b:webvoyager --harness eve --tool stagehand_facade -l 1 -t 1 -e browserbase), run twice (before/after the budget fix): 5 and 3 Stagehand tool calls via the facade, final answer produced.

Stacked on the pi PR. Implemented with Codex (gpt-5.6) under supervision; Claude + codex exec review findings (abort cancel, unredacted logs, unbounded listTools/cleanup, tmpdir leak, budget off-by-one) fixed in-branch.


Summary by cubic

Adds an Eve external harness to evals via a new private @browserbasehq/stagehand-integrations-eve-sdk, enabling evals run ... --harness eve. Eve wasn't supported before; now tasks run with MCP-bridged tools, correct cancellation, redacted transcripts, and budget counted only on completed tool results.

Implementation

  • Session layer spawns eve dev per task (no UI, random port), streams NDJSON, extracts usage/cost, and on abort cancels the Eve session exactly once and kills the dev server.
  • Tool adapter generates a temporary Eve agent app whose tools proxy to MCP servers, with bounded connect/listTools/close and cleanup timeouts, late-client close, and mkdtemp rollback.
  • Runner maps Eve events to normalized tool calls, usage, image evidence, and metrics, and registers "eve" with default model openai/gpt-5.4-mini.
  • Non-browser framework tools like load_skill are disabled; the Eve SDK is lazy-loaded to avoid import-time side effects.
  • Adds the package to Turbo/CI/Vitest with tests covering abort/kill/redaction, tool-adapter prep/cleanup/timeouts, mkdtemp rollback, lazy-load, and registry updates.

Rollout

  • Requires Node 24+; provide MCP mounts via STAGEHAND_EVE_MCP_SERVERS (JSON array).
  • Run with evals run b:<dataset> --harness eve --tool <tool> -e browserbase.
  • Optional timeouts: EVAL_EVE_MCP_LIST_TOOLS_TIMEOUT_MS (default 60s) and EVAL_AGENT_MOUNT_CLEANUP_TIMEOUT_MS.
  • Configure models with EVAL_EVE_MODELS; defaults to openai/gpt-5.4-mini.