feat(evals): port HardBenchmark suite onto the harness wave
Draft
MI
1mo ago+17360+17360+01234567890123456789012345678901234567890123456789-344-344-012345678901234567890123456789
Stacked on #2812 (top of the harness wave). Part of the harness×model campaign.
What
Ports the 46-task HardBenchmark suite — the residual hard core of WebTailBench + Online-Mind2Web (tasks both frontier models failed, audited to genuine model faults, date-rotted tasks dropped) — onto the wave head.
datasets/hardbenchmark/HardBenchmark_data.jsonl(verbatimprecomputed_rubricper row + audited provenance:source_suite,failure_mode,capability_axis)suites/hardbenchmark.tsbuilder; registered asagent/hardbenchmarkin discovery, planner, external-harness plan, TUI (b:hardbenchmark), config- Provenance fields added to testcase metadata types so regressions attribute to a capability, not just a task
Notes
- Rubric-carrying: no LLM-generated rubric divergence between harnesses.
- The validity audit (quarantine flags, review list) lands in the follow-up PR on
feat/facade-batch-surface.
Summary by cubic
Adds the 46-task HardBenchmark residual suite to the eval harness to focus model evaluation on audited failures from WebTailBench and Online-Mind2Web. Unlike the parent suites, it runs all 46 cases by default and preserves each task’s precomputed rubric.
Included
- Registers
agent/hardbenchmarkacross discovery, planning, external harnesses, config, and theb:hardbenchmarkTUI shorthand. - Supports task ID selection, sampling, failure-mode filtering, and suite-specific limits.
- Adds source suite, failure mode, and capability axis to testcase metadata for regression analysis.