evals: ground rubric grading in harness-observed task artifacts
Draft
SH
3mo ago+224+224+012345678901234567890123456789-10-10-01234567890123456789
Part of STG-2671 (LLMJ grading — LLM grades only, never drives).
Grounds rubric grading in harness-observed task artifacts. Code tool surfaces stringify run-tool results, so the verifier's final-screenshot anchor was empty on every v4_code/playwright_code/cdp_code run — grading rested on the agent's textual self-report (the verifier itself emitted a blocking trajectory_capture finding asking for exactly this).
What's inside
claudeCodeRunner— after the agent finishes, before cleanup: capture terminal page state through the tool surface (bounded, best-effort), thread into the trajectoryclaudeCodeAdapter— final-observation anchor prefers the harness-observed artifact (screenshot + URL) over agent-returned images; browse_cli fallback preservedrubricCache— never cache or serve a zero-item rubric: an empty cached rubric silently disabled grading for every trial of a task in both comparison arms (observed live onAllrecipes--2).gitignore— exclude runtime.rubric-cache/
Verification
- Unit tests pin the anchor preference order and both rubric guards
- Poisoned cache entry purged; framework suite green
Stacked on stg-2671-v4-harness.
Summary by cubic
Ground rubric grading in harness-observed artifacts so the verifier uses a real final screenshot and URL for v4_code/playwright_code/cdp_code. Also guard the rubric cache against empty rubrics that disabled grading; part of STG-2671.
-
New Features
claudeCodeRunner: capture terminal page state (screenshot + URL) via the tool surface after the agent finishes, and pass it into the trajectory.claudeCodeAdapter: prefer the harness terminal artifact for the final observation; fall back to the latest agent-returned image (e.g.,browse_cli).
-
Bug Fixes
rubricCache: never read or persist empty rubrics; treat as a cache miss and regenerate.- Tests: add coverage for anchor preference and empty-rubric guards; ignore
.rubric-cache/.