feat(evaluator): standalone stagehand-evaluator package
MI
1d ago+9601+9601+0123456789012345678901234567890123456789-2-2-0123456789
Summary
Adds @browserbasehq/stagehand-evaluator: a browser-free rubric verifier extracted from the pinned stagehand-v3@3.7.1 verifier and rebuilt on a direct AI SDK judge. Evals still uses the v3 verifier after this PR; the rewire is the next PR in this stack.
Stack: 1/2 · base main → this → #3100
What's in the package
- Faithful extraction. Ported from the 3.7.1 source maps (hashes in
UPSTREAM.json) with a recorded-response parity test against the pinned module. - Evidence. Chunked retrieval over every recorded observation; answer anchors locate claimed exact values (quotes, prices, identifiers) anywhere in the trajectory; tool-derived agent text is tagged as observation; embedded MCP screenshots recovered; screenshot provenance labels.
- Judgment. Three-way outcome state (supported / contradicted / unresolved) with explicit critical-point, identity/constraint and deliverable checks; deterministic answer-structure classification (tables, delimited records, JSON) and rubric fallback-clause scan.
- Diagnostics. Versioned
health,execution(termination reason, issues, browser-use rule),failureClass. A dead or failing judge is reported as ungraded, never as a negative reward. - Reliability. Judge health check once per model per process; 60 s per-attempt timeout with one retry.
- Opt-in knobs (defaults unchanged): image selection
llm | recency | signals, thinking budget, local OCR evidence, requirement checklist, relaxed decision rule. See the README table. scripts/replay.mjsreplays frozen manifests with build, config and rubric stamps.
Measured (full protocol kept locally)
Pooled held-out set, 223 rows labeled blind by two independent graders, gemini-3.8-flash judge:
| Evaluator | FP | FN | F1 | LLM calls/row | Median s |
|---|---|---|---|---|---|
| Stack V3 verifier (current) | 6 | 19 | 0.880 | ~20 | ~50 |
This package, signals + thinking cap | 5 | 9 | 0.936 | 1 | ~8.4 |
Paired vs the current verifier: 18–7 discordant rows (McNemar p = 0.04). The recommended screenshot-selection settings are not turned on in this stack; that is a separate decision.
Notes for reviewers
pnpm-lock.yaml: adds only the package's own dependencies, at versions the workspace already resolves.turbo.json: registers the package'sbuildandtypechecktasks.ci.yml: uploadspackages/evaluator/distwith the other build artifacts so downstream jobs can import it.- Direct judge providers:
google/…andopenai/…; others via an injectedLLMClient.
Testing
pnpm --filter @browserbasehq/stagehand-evaluator build,typecheck: clean- Evaluator unit tests: 49 passed, 1 skipped
oxfmt --checkclean on changed files;oxlintno errors