feat(evals): add eve bench harness (integrations/eve-sdk)
Merged
MI
2mo ago+3193+3193+0123456789012345678901234567890123456789-28-28-01234567890123456789
What
Adds --harness eve to evals, backed by a new private @browserbasehq/stagehand-integrations-eve-sdk package.
packages/integrations/eve-sdk—runEveSession: eve 0.29.4 has no in-process run-agent SDK and no stdio MCP support, so the session layer spawnseve dev --no-ui --port 0per row against a generated agent app and consumes the HTTP NDJSON event stream (actions.requested/action.result/ turn boundaries); tool-step budget counted on completed results; abort →session.cancel()+ dev-server SIGTERM; redaction.packages/evals/framework/eveToolAdapter.ts— generates the eve agent app (agent/tools/<server>__<tool>.tsproxies over an@modelcontextprotocol/sdkstdio client) forvia:"mcp"mounts; boundedlistTools/cleanup; mkdtemp rollback;load_skilland other non-browser framework tools disabled.packages/evals/framework/harnesses/eveAdapter.ts— events →NormalizedToolCall[].- Registered via
defineExternalHarness. Requires Node ≥ 24 (eve's own floor).
Testing
- Full unit gate green (evals + eve-sdk 16 + all other suites).
- Connected smoke (
evals run b:webvoyager --harness eve --tool stagehand_facade -l 1 -t 1 -e browserbase), run twice (before/after the budget fix): 5 and 3 Stagehand tool calls via the facade, final answer produced.
Stacked on the pi PR. Implemented with Codex (gpt-5.6) under supervision; Claude + codex exec review findings (abort cancel, unredacted logs, unbounded listTools/cleanup, tmpdir leak, budget off-by-one) fixed in-branch.
Summary by cubic
Adds an Eve external harness to evals via a new private @browserbasehq/stagehand-integrations-eve-sdk, enabling evals run ... --harness eve. Eve wasn't supported before; now tasks run with MCP-bridged tools, correct cancellation, redacted transcripts, and budget counted only on completed tool results.
Implementation
- Session layer spawns
eve devper task (no UI, random port), streams NDJSON, extracts usage/cost, and on abort cancels the Eve session exactly once and kills the dev server. - Tool adapter generates a temporary Eve agent app whose tools proxy to MCP servers, with bounded connect/listTools/close and cleanup timeouts, late-client close, and
mkdtemprollback. - Runner maps Eve events to normalized tool calls, usage, image evidence, and metrics, and registers
"eve"with default modelopenai/gpt-5.4-mini. - Non-browser framework tools like
load_skillare disabled; the Eve SDK is lazy-loaded to avoid import-time side effects. - Adds the package to Turbo/CI/Vitest with tests covering abort/kill/redaction, tool-adapter prep/cleanup/timeouts,
mkdtemprollback, lazy-load, and registry updates.
Rollout
- Requires Node 24+; provide MCP mounts via
STAGEHAND_EVE_MCP_SERVERS(JSON array). - Run with
evals run b:<dataset> --harness eve --tool <tool> -e browserbase. - Optional timeouts:
EVAL_EVE_MCP_LIST_TOOLS_TIMEOUT_MS(default 60s) andEVAL_AGENT_MOUNT_CLEANUP_TIMEOUT_MS. - Configure models with
EVAL_EVE_MODELS; defaults toopenai/gpt-5.4-mini.