Keep verifier outcomes, uncertainty, and execution failures auditable
Separates verified task completion from execution health. Verification errors fail closed, preserve the agent report separately, and retain captured evidence and raw uncertainty for audit. A supported completion can still pass after a late browser disconnect.
Preserves judge verdicts beside deterministic evidence gates; missing criterion scores remain in the denominator. Blocker wording and execution error alone do not erase a completed rubric requirement. Numeric grounding remains advisory by default. It includes captured step and terminal accessibility text, records terminal matches separately, and treats unknown-page text as untrusted. Batch summaries expose explicit zero-browser passes; the configured verifiability gate rejects them. Producer-dependent usage, cost, facade and lifecycle fields are documented as conditional.
Offline verification uses the same model factory as live grading: CLI override, then EVAL_VERIFIER_MODEL, then the shared default. Thrown verification errors and uncertain judgments are preserved as explicit ungraded artifacts without top-level outcome/process scores, including live persistence and tracing; the offline command exits unsuccessfully. Trace labels are restricted to filename components, and failed writes emit sanitized warnings.
Judge policy still needs confirmation: this proposed change selects google/gemini-3.5-flash by default, while current main uses 2.5 Flash. The completed 108-case compatibility gate explicitly used 2.5 Flash and does not validate the proposed 3.5 default.
Validation: 104 focused verifier/CLI/span/persistence/summary tests at this boundary, eval typecheck and scoped formatting; 13 assertions reproduced the new review gaps before fixes. No frozen 108-case compatibility inputs changed. No evaluator-v4 implementation.
Stack position: 9/11. Base: evals/consolidation-10-usage-provenance. Each PR contains the change relative to its immediate predecessor.
Reviewer entry points:
packages/evals/framework/verifierAdapter.tspackages/evals/framework/verifierGates.tspackages/evals/docs/verifier-gates.md
Reordered stack: #2891 → #2892 → #2895 → #2896 → #2897 → #2898 → #2899 → #2900 → #2901 → #2893 → #2894.
Readiness: CI passed on af1cf70392. The judge-model choice remains an open review decision.