Keep verifier outcomes, uncertainty, and execution failures auditable

Merged
MI
2mo ago
+2506-75

Separates verified task completion from execution health. Verification errors fail closed, preserve the agent report separately, and retain captured evidence and raw uncertainty for audit. A supported completion can still pass after a late browser disconnect.

Preserves judge verdicts beside deterministic evidence gates; missing criterion scores remain in the denominator. Blocker wording and execution error alone do not erase a completed rubric requirement. Numeric grounding remains advisory by default. It includes captured step and terminal accessibility text, records terminal matches separately, and treats unknown-page text as untrusted. Batch summaries expose explicit zero-browser passes; the configured verifiability gate rejects them. Producer-dependent usage, cost, facade and lifecycle fields are documented as conditional.

Offline verification uses the same model factory as live grading: CLI override, then EVAL_VERIFIER_MODEL, then the shared default. Thrown verification errors and uncertain judgments are preserved as explicit ungraded artifacts without top-level outcome/process scores, including live persistence and tracing; the offline command exits unsuccessfully. Trace labels are restricted to filename components, and failed writes emit sanitized warnings.

Judge policy still needs confirmation: this proposed change selects google/gemini-3.5-flash by default, while current main uses 2.5 Flash. The completed 108-case compatibility gate explicitly used 2.5 Flash and does not validate the proposed 3.5 default.

Validation: 104 focused verifier/CLI/span/persistence/summary tests at this boundary, eval typecheck and scoped formatting; 13 assertions reproduced the new review gaps before fixes. No frozen 108-case compatibility inputs changed. No evaluator-v4 implementation.

Stack position: 9/11. Base: evals/consolidation-10-usage-provenance. Each PR contains the change relative to its immediate predecessor.

Reviewer entry points:

Reordered stack: #2891 → #2892 → #2895 → #2896 → #2897 → #2898 → #2899 → #2900 → #2901 → #2893 → #2894.

Readiness: CI passed on af1cf70392. The judge-model choice remains an open review decision.