feat(evals): grade with standalone stagehand-evaluator
MI
1d ago+763+763+012345678901234567890123456789-167-167-012345678901234567890123456789
Summary
Switches evals grading from the stagehand-v3 V3Evaluator carrier to the standalone @browserbasehq/stagehand-evaluator package from #3099, and carries over the structured health and gate semantics.
Stack: 2/2 · base #3099 → this
Changes
verifierAdapter/evals verifybuild the judge from the evaluator package; no V3 object is constructed. Model selection still goes throughverifierModel.ts(--model>EVAL_VERIFIER_MODEL> default), so the run header reports the judge actually used; an explicit override fails loud when its provider key is missing.- Default judge:
google/gemini-3.5-flash→google/gemini-3.8-flash, the model the evaluator was calibrated on.verifierModel.tskeeps it as a literal (it is dependency-free for the CLI header) and a new test pins it to the evaluator'sDEFAULT_VERIFIER_MODEL. Tests that print the default judge (doctor probe, run header) are updated to match. benchHarnessvalidates the judge once before preparing a tool adapter, so a dead judge never allocates a browser.verifierGates: accepts v3-style verdicts and structured outcome states; a proven completion survives a later disconnect; browser use is keyed on recorded server identity with a documented name fallback.- A failed verification is ungraded (no outcome/process score) with
healthand the agent's own report preserved, matching the stack's runner contract. verifierTraceand tests import types/mocks from the evaluator package.- The
stagehand-v3pin stays for non-verifier consumers.
Testing
turbo run build --filter=@browserbasehq/stagehand-evals...: 17/17turbo run typecheck(evals + evaluator): 18/18- Evals unit tests: 1225 passed (130 files); evaluator: 49 passed, 1 skipped
oxfmt --checkclean on changed files;oxlintno errors (twono-base-to-stringwarnings are on lines unchanged from main)