evals: ground rubric grading in harness-observed task artifacts

Draft
SH
3mo ago
+224-10

Part of STG-2671 (LLMJ grading — LLM grades only, never drives).

Grounds rubric grading in harness-observed task artifacts. Code tool surfaces stringify run-tool results, so the verifier's final-screenshot anchor was empty on every v4_code/playwright_code/cdp_code run — grading rested on the agent's textual self-report (the verifier itself emitted a blocking trajectory_capture finding asking for exactly this).

What's inside

  • claudeCodeRunner — after the agent finishes, before cleanup: capture terminal page state through the tool surface (bounded, best-effort), thread into the trajectory
  • claudeCodeAdapter — final-observation anchor prefers the harness-observed artifact (screenshot + URL) over agent-returned images; browse_cli fallback preserved
  • rubricCache — never cache or serve a zero-item rubric: an empty cached rubric silently disabled grading for every trial of a task in both comparison arms (observed live on Allrecipes--2)
  • .gitignore — exclude runtime .rubric-cache/

Verification

  • Unit tests pin the anchor preference order and both rubric guards
  • Poisoned cache entry purged; framework suite green

Stacked on stg-2671-v4-harness.


Summary by cubic

Ground rubric grading in harness-observed artifacts so the verifier uses a real final screenshot and URL for v4_code/playwright_code/cdp_code. Also guard the rubric cache against empty rubrics that disabled grading; part of STG-2671.

  • New Features

    • claudeCodeRunner: capture terminal page state (screenshot + URL) via the tool surface after the agent finishes, and pass it into the trajectory.
    • claudeCodeAdapter: prefer the harness terminal artifact for the final observation; fall back to the latest agent-returned image (e.g., browse_cli).
  • Bug Fixes

    • rubricCache: never read or persist empty rubrics; treat as a cache miss and regenerate.
    • Tests: add coverage for anchor preference and empty-rubric guards; ignore .rubric-cache/.