feat(evals): add Cursor agent bench harness (integrations/cursor-sdk)

Merged
MI
2mo ago
+2043-12

What

Adds --harness cursor to evals, backed by a new private @browserbasehq/stagehand-integrations-cursor-sdk package, plus a minimal packages/integrations/cursor facade example (mcp.json + README, fx-package shape). The Cursor agent is CLI-only (agent, legacy cursor-agent; no npm SDK).

  • packages/integrations/cursor-sdk — runCursorSession: spawns agent -p --output-format stream-json with a project-local .cursor/mcp.json pointing at the stagehand facade stdio server, parses the stream into tool-call/result/usage events, abort → child kill, redaction, status/stopReason normalization.
  • packages/evals/framework/cursorToolAdapter.ts — via:"mcp" mounts only (stagehand_facade first, playwright_mcp, chrome_devtools_mcp; browse_cli/code surfaces rejected).
  • packages/evals/framework/harnesses/cursorAdapter.ts — events → NormalizedToolCall[].
  • Registered via defineExternalHarness on the shared external-runner skeleton.

Testing

  • Full unit gate green (evals + cursor-sdk 8 + all other suites).
  • Connected smoke not yet run: the agent CLI is not installed on the dev machine (curl https://cursor.com/install -fsS | bash). Command when available: evals run b:webvoyager --harness cursor --tool stagehand_facade -l 1 -t 1 -e browserbase.

Stacked on the fx PR; top of the harness stack. Implemented with Codex (gpt-5.6) under supervision.

Summary by cubic

Adds a registered Cursor agent CLI harness so evals can run with --harness cursor over MCP-only mounts, without launching a browser or modifying files. The Cursor CLI reports no token usage, so token metrics are zero.

  • Supported tool surfaces: stagehand_facade (default), playwright_mcp, chrome_devtools_mcp; other surfaces (e.g., browse_cli, code mounts) are rejected.
  • Registration: cursor is in the bench harness registry with default model cursor/auto; wired into CI, turbo, and vitest with unit tests for registry, runner, adapter, and SDK.
  • Process/workspace: writes a temp .cursor/mcp.json for active MCP servers, inherits env, and cleans up. Override the binary with EVAL_CURSOR_AGENT_PATH; toggle sandbox via EVAL_CURSOR_SANDBOX=enabled|disabled; cap completed tool steps via EVAL_CURSOR_MAX_STEPS or AGENT_EVAL_MAX_STEPS (default 50).
  • CLI invocation: runs agent -p --output-format stream-json --force --trust --approve-mcps; API keys are never passed on the command line, and stream output is sanitized for secrets.
  • Metrics: cursor_input_tokens, cursor_output_tokens, cursor_total_tokens (always zero), cursor_duration_ms, cursor_tool_steps.
  • Trajectory mapping: assistant text becomes reasoning; started/completed tool_call pairs become steps, and started calls without a completed envelope (abort/kill mid-call) fail closed; MCP images become evidence with a final-observation fallback; step observations attach by MCP tool ordinal. Results are parsed strictly first, then leniently for fenced/prose JSON.
  • New private package @browserbasehq/stagehand-integrations-cursor-sdk adds the process runner and parsers; packages/integrations/cursor provides an mcp.json template and AGENTS.md guidance.
  • Run locally: install the Cursor CLI (curl https://cursor.com/install -fsS | bash), set CURSOR_API_KEY, then e.g. evals run b:webvoyager --harness cursor --tool stagehand_facade -l 1 -t 1 -e browserbase.