feat(evals): add stagehand_facade tool surface

Merged
MI
3mo ago
+358-21

Stacked on the codex-sdk extraction PR. Part 4 (final) of the harness consolidation stack — this closes the loop: evals now benchmarks the byte-identical facade surface the claude-code/codex/pi integrations ship.

What

New via:"mcp" tool surface stagehand_facade: the mount spawns the shipped facade stdio server (@browserbasehq/stagehand-integrations/facade/stdio-server) with an allowlisted STAGEHAND_*/BROWSERBASE_* env (browser selection forced to match the eval environment) and FACADE_AGENT_INSTRUCTIONS by identity. Registered for both external harnesses, selectable alongside stagehand_code (not replacing it). The facade server owns its browser (tool_launch_local/tool_create_browserbase); evidence semantics match the other external-MCP surfaces (verification via the tool_result stream). Also ignores evals run artifacts (.trajectories/, rubric cache) — generated output with session IDs that was dirtying trees.

Verification

  • Full gates ✅; surface test pins mount shape, prompt identity, env filtering, and harness registration
  • End-to-end: evals run b:webvoyager --harness claude_code --tool stagehand_facade -l 1 -e browserbase → 3/3 trials complete, agents drove mcp__stagehand__{run,snapshot,screenshot}, 2/3 graded pass, 0/12 criteria unverifiable (better verifiability than the handles surface)

Summary by cubic

Adds stagehand_facade, an MCP tool surface that launches the shipped facade stdio server so evals benchmark the exact surface integrations ship. The facade owns its browser, verification uses the tool_result stream, and it's selectable alongside stagehand_code for the agent harnesses rather than replacing it.

  • stagehand_facade is mount-only: left out of the core tool list and TUI help since its runner-side session throws on every page operation, but resolvable for the claude_code and codex harness mounts.
  • The mount spawns the stdio server with FACADE_AGENT_INSTRUCTIONS and an allowlisted env, forces STAGEHAND_BROWSER by environment, and applies longer MCP timeouts in the Codex config.
  • Mount cleanup is best-effort; the stdio child and browser belong to the agent harness process tree, with Browserbase session TTL bounding the remote leak case.
  • TUI help now lists stagehand_code, which was previously missing from the valid core tools list.

Review updates (2026-08-29)

  • Mount-only: stagehand_facade no longer appears in listCoreTools() or the TUI help — its CoreSession throws on every page operation, so core-tier selection failed deterministically. It stays resolvable via getCoreTool for the agent harness mounts.
  • Cleanup limitation documented: the facade stdio child (and its browser) belongs to the agent harness process tree; evals-side cleanup is best-effort and cannot reap it (Browserbase session TTL bounds the remote case).