Allow expected-failures entries to name a single check

Merged
FE
4mo ago
+564-95

Closes #404.

Motivation and Context

An entry names a scenario, but a scenario is many checks. server-stateless is 29. Spec PR #3002 (picked up here in #403) breaks 2 of them, so today the only way to stay green is:

# expected-failures.yaml — before
server:
  - server-stateless

That excuses all 29. The other 27 stop being enforced until the entry comes out. typescript-sdk#2513 has exactly this entry, with a comment listing by hand which checks are really affected.

Now you can name them:

# expected-failures.yaml — after
server:
  - server-stateless:sep-2575-request-meta-client-info-optional
  - server-stateless:sep-2575-server-identifies-in-result-meta

27 checks stay enforced. Anything else in server-stateless that breaks is still an unexpected failure.

Both forms work, and they mix:

server:
  - tasks-lifecycle                       # whole scenario, as before
  - server-stateless:sep-2575-server-implements-discover

The check id is the column the runner already prints, so you copy it out of the failing run.

Two configs are rejected rather than silently misread:

server:
  - server-stateless: sep-2575-foo   # space makes this a YAML mapping, not a string
  - server-stateless                 # contradicts the per-check entry below
  - server-stateless:sep-2575-bar

How Has This Been Tested?

Against the v2 typescript-sdk server, --scenario server-stateless --spec-version 2026-07-28:

[sep-2575-request-meta-client-info-optional] FAILURE
[sep-2575-server-identifies-in-result-meta ] WARNING
Passed: 28/29, 1 failed, 1 warnings

Naming both exits 0. Naming one exits 1 on the other:

Expected failures (in baseline):
  ~ server-stateless:sep-2575-request-meta-client-info-optional
Unexpected failures (not in baseline):
  ✗ server-stateless:sep-2575-server-identifies-in-result-meta

To show what the coarse entry costs, I broke an unrelated fixture the scenario depends on: - server-stateless exits 0 and hides the regression, the per-check baseline exits 1 and names it.

Plus unit tests for stale-per-check, repeated ids, and both rejected configs.

Breaking Changes

None for baseline files — YAML parses - a:b as a plain string, so existing baselines and the action input are untouched.

loadExpectedFailures now returns parsed BaselineEntry[] rather than string[]. Internal only: the package ships a CLI (bin, no main/exports), and src/index.ts needs no change.

Types of changes

  • New feature (non-breaking change which adds functionality)

Checklist

  • I have read the MCP Documentation
  • My code follows the repository's style guidelines
  • New and existing tests pass locally
  • I have added appropriate error handling
  • I have added or updated documentation as needed

Additional context

A check id matches every occurrence of that id, since ids repeat within a run. Occurrences collapse to one verdict, most-severe first, so baselining a repeated id excuses all of them — coarser than ideal, still far narrower than the scenario. Absent and SKIPPED checks are tolerated, matching how the runner already treats them as green.

loadExpectedFailures owns the entry grammar and evaluateBaseline only matches, so rejecting a bad entry and interpreting a good one don't live in separate layers.

collapseDuplicateChecks moves from scenarios/client/auth/dpop.ts to src/checks/collapse.ts with its tests — expected-failures.ts needs the reducer and can't pull in dpop.ts's auth server stack. No behavior change.