evals: ground rubric grading in harness-observed task artifacts

Closed
SH
3mo ago
+1467-609

Nondeterministic-suite stack 2/4, on #2472's base. Supersedes #2389.


Summary by cubic

Ground rubric-based grading in harness-captured terminal artifacts and guard the rubric cache from empty entries. This restores final screenshot anchors for v4_code/playwright_code/cdp_code runs and prevents silent zero-scoring in non-deterministic v4 evals (STG-2671).

  • New Features

    • Capture the final page state (screenshot + URL) via the tool surface after the agent finishes and attach it to the trajectory.
    • Prefer this harness artifact as the verifier’s final observation; fall back to the last agent-returned image if needed.
  • Bug Fixes

    • rubricCache no longer reads or writes empty rubrics; treats them as cache misses to avoid grading being disabled.
    • Add unit tests for artifact preference and empty-rubric guards; ignore .rubric-cache/ in git.