feat(evals): port HardBenchmark suite onto the harness wave

Draft
MI
1mo ago
+17360-344

Stacked on #2812 (top of the harness wave). Part of the harness×model campaign.

What

Ports the 46-task HardBenchmark suite — the residual hard core of WebTailBench + Online-Mind2Web (tasks both frontier models failed, audited to genuine model faults, date-rotted tasks dropped) — onto the wave head.

  • datasets/hardbenchmark/HardBenchmark_data.jsonl (verbatim precomputed_rubric per row + audited provenance: source_suite, failure_mode, capability_axis)
  • suites/hardbenchmark.ts builder; registered as agent/hardbenchmark in discovery, planner, external-harness plan, TUI (b:hardbenchmark), config
  • Provenance fields added to testcase metadata types so regressions attribute to a capability, not just a task

Notes

  • Rubric-carrying: no LLM-generated rubric divergence between harnesses.
  • The validity audit (quarantine flags, review list) lands in the follow-up PR on feat/facade-batch-surface.

Summary by cubic

Adds the 46-task HardBenchmark residual suite to the eval harness to focus model evaluation on audited failures from WebTailBench and Online-Mind2Web. Unlike the parent suites, it runs all 46 cases by default and preserves each task’s precomputed rubric.

Included

  • Registers agent/hardbenchmark across discovery, planning, external harnesses, config, and the b:hardbenchmark TUI shorthand.
  • Supports task ID selection, sampling, failure-mode filtering, and suite-specific limits.
  • Adds source suite, failure mode, and capability axis to testcase metadata for regression analysis.