feat(evals): grade with standalone stagehand-evaluator

MI
1d ago
+763-167

Summary

Switches evals grading from the stagehand-v3 V3Evaluator carrier to the standalone @browserbasehq/stagehand-evaluator package from #3099, and carries over the structured health and gate semantics.

Stack: 2/2 · base #3099 → this

Changes

  • verifierAdapter / evals verify build the judge from the evaluator package; no V3 object is constructed. Model selection still goes through verifierModel.ts (--model > EVAL_VERIFIER_MODEL > default), so the run header reports the judge actually used; an explicit override fails loud when its provider key is missing.
  • Default judge: google/gemini-3.5-flash → google/gemini-3.8-flash, the model the evaluator was calibrated on. verifierModel.ts keeps it as a literal (it is dependency-free for the CLI header) and a new test pins it to the evaluator's DEFAULT_VERIFIER_MODEL. Tests that print the default judge (doctor probe, run header) are updated to match.
  • benchHarness validates the judge once before preparing a tool adapter, so a dead judge never allocates a browser.
  • verifierGates: accepts v3-style verdicts and structured outcome states; a proven completion survives a later disconnect; browser use is keyed on recorded server identity with a documented name fallback.
  • A failed verification is ungraded (no outcome/process score) with health and the agent's own report preserved, matching the stack's runner contract.
  • verifierTrace and tests import types/mocks from the evaluator package.
  • The stagehand-v3 pin stays for non-verifier consumers.

Testing

  • turbo run build --filter=@browserbasehq/stagehand-evals...: 17/17
  • turbo run typecheck (evals + evaluator): 18/18
  • Evals unit tests: 1225 passed (130 files); evaluator: 49 passed, 1 skipped
  • oxfmt --check clean on changed files; oxlint no errors (two no-base-to-string warnings are on lines unchanged from main)