evals: ground rubric grading in harness-observed task artifacts
Closed
SH
3mo ago+1467+1467+0123456789012345678901234567890123456789-609-609-012345678901234567890123456789
Nondeterministic-suite stack 2/4, on #2472's base. Supersedes #2389.
Summary by cubic
Ground rubric-based grading in harness-captured terminal artifacts and guard the rubric cache from empty entries. This restores final screenshot anchors for v4_code/playwright_code/cdp_code runs and prevents silent zero-scoring in non-deterministic v4 evals (STG-2671).
-
New Features
- Capture the final page state (screenshot + URL) via the tool surface after the agent finishes and attach it to the trajectory.
- Prefer this harness artifact as the verifier’s final observation; fall back to the last agent-returned image if needed.
-
Bug Fixes
rubricCacheno longer reads or writes empty rubrics; treats them as cache misses to avoid grading being disabled.- Add unit tests for artifact preference and empty-rubric guards; ignore
.rubric-cache/in git.