chore(evals): evaluator calibration scripts and notes

Draft
MI
19d ago
+1451-0

Summary

The measurement pipeline used to calibrate the evaluator, plus EVALUATOR_V4_NOTES.md with the full protocol and results. No runtime code; safe to drop from the stack if we'd rather keep this out of the repo.

Stack: 3/3 · #3099 → #3100 → this

Contents

  • fresh-corpus-sample.py: task-split, verdict-balanced sampler with per-model and per-task caps
  • label-blind.py: blind outcome labels from two independent grader agents (codex / claude backends) under the HardBench v1.2 policy; graders never see a verdict
  • fresh-score.py: F1, FPR, FNR, judge calls, tokens and latency per replay arm
  • Earlier-phase reconciliation scripts: evaluator-baseline, evaluator-recut, label-overlay, two-vote-overlay, v12-relabel, rubric-clarify
  • hardbench-audit-luna.py: per-row audit driver (adds --only-runs)

Headline from the notes

Per-screenshot relevance scoring was 83–92% of the verifier's LLM calls. Replacing it with a zero-LLM selector built from trajectory signals (answer anchors, state-changing actions, page entry/exit, text-poor steps) gives F1 0.936 vs 0.880 for the current verifier on 223 blind-labeled held-out rows, at 1 call and ~8 s per row instead of ~20 calls and ~50 s.

Testing

  • All scripts py_compile; test_evaluator_baseline.py passes

Summary by cubic

Adds the measurement pipeline used to calibrate the evaluator, plus EVALUATOR_V4_NOTES.md documenting the full protocol and results. No runtime code — this is safe to drop if we'd rather not keep tooling in the repo.

Headline result

  • Per-screenshot relevance scoring drove 83–92% of the verifier's LLM calls.
  • Replacing it with a zero-LLM selector built from trajectory signals (answer anchors, state-changing actions, page entry/exit, text-poor steps) gives F1 0.936 vs 0.880 for the current verifier on 223 blind-labeled held-out rows, at 1 call and ~8 s per row instead of ~20 calls and ~50 s.

Contents

  • fresh-corpus-sample.py, label-blind.py, and fresh-score.py form the reproducible sample → blind-label → score pipeline with per-model and per-task caps.
  • Earlier-phase reconciliation scripts: evaluator-baseline, evaluator-recut, label-overlay, two-vote-overlay, v12-relabel, rubric-clarify.
  • hardbench-audit-luna.py adds a per-row audit driver (--only-runs).
  • All scripts py_compile; test_evaluator_baseline.py passes.