fix: fail checks whose prerequisite is missing instead of skipping them

Merged
FE
5mo ago
+902-117

A check whose prerequisite is missing — the server under test lacks a diagnostic fixture tool, or advertises a feature it then rejects — now fails with a Not testable: cause instead of reporting SKIPPED. SKIPPED is excluded from pass/fail counts, exit codes, and the expected-failures baseline, so these gaps read as green: a server could dodge the entire SEP-2575 capability-constraint and subscription surface (7 MUST-level checks) and still exit 0. This implements the policy direction from #248: a scenario that can't execute counts as a failure; the escape hatches are scenario filtering and the expected-failures baseline, not in-scenario skipping.

Motivation and Context

Concretely, before this change, a draft server with no diagnostic tools that advertises tools.listChanged but rejects subscriptions/listen:

sep-2575-server-rejects-undeclared-capability      SKIPPED
sep-2575-missing-capability-http-400               SKIPPED
sep-2575-http-server-no-independent-requests…      SKIPPED
sep-2575-server-no-log-without-loglevel            SKIPPED
sep-2575-server-sends-subscription-ack             SKIPPED
sep-2575-server-tags-subscription-id               SKIPPED
sep-2575-server-honors-notification-filter         SKIPPED
Passed: 10/21, 11 failed

The seven SKIPPED MUSTs vanish from the denominator, don't affect the exit code, and evaluateBaseline treats them as passing — worse, a baselined failing scenario whose checks degrade to SKIPPED (the server drops a fixture tool) flips its baseline entry to "stale — remove", which reads as fixed. After:

Passed: 10/28, 18 failed, 1 warnings
  Error: Not testable: server does not list the diagnostic tool 'test_missing_capability' in tools/list and the probe call did not exercise it (required for the undeclared-capability rejection)
  Error: Not testable: server advertises listChanged/subscribe capabilities but answers subscriptions/listen with -32601 (Method not found)
  …

Every unexercised requirement is red with its cause named; the one remaining SKIPPED is the legitimately-inapplicable case (prompts.listChanged never declared).

What changed

New src/scenarios/untestable.ts — notTestable(reason) formats the stable Not testable: errorMessage prefix and untestableCheck(...) builds a failing check with details.untestable: true, so reports and dashboards can distinguish "violated" from "could not be verified" without a new status enum. Severity follows the underlying requirement's keyword (MUST → FAILURE, SHOULD → WARNING).

server-stateless:

  • The capability-constraint pair now distinguishes a fixture gap from a genuine violation, strongest evidence first: a result from the probe call proves the tool executed → genuine MUST failure regardless of whether tools/list mentioned it; a non--32021 error from a listed tool → wrong-code failure; an error from an unlisted tool → untestable failure naming the tool. The tools/list snapshot is fetched once and reused by the capabilities-match check (which now reports its own probe failure instead of misattributing it).
  • The streaming/logging probes (test_streaming_elicitation, test_logging_tool) fail with the missing tool named instead of skipping.
  • The three subscription checks are now capability-aware: -32601 on subscriptions/listen is a legitimate SKIP only when an observed server/discover advertisement shows no listChanged/subscribe capability. A server that advertises them but rejects the method fails (it claims a feature it does not serve), and when discover itself was never observed the gap is untestable rather than silently excused.
  • The trigger-hook gates on the two list-changed SHOULD checks report untestable WARNINGs (severity follows the keyword) instead of skipping. The capability-undeclared gates on those same checks stay SKIPPED — that's the genuinely optional case.
  • The sep-2575-http-server-error-jsonrpc-id SUCCESS backfill is now conditional on at least one error response having been observed (counted at the sendRpc transport seam, so every probe contributes); a server that wrongly accepted every invalid probe no longer collects a vacuous pass for the id-echo requirement.

SEP-2243 server scenarios:

  • http-custom-header-server-validation: the no-x-mcp-header-tool and no-string-param gates emit untestable FAILUREs (gate row + all four declared requirement IDs) instead of an all-SKIPPED run that previously passed with zero coverage of four MUSTs. The declared-ID backfill keeps the emitted ID set stable for traceability.
  • http-header-validation: the three Mcp-Name cases (mismatch-reject, missing-reject, whitespace-accept) were previously omitted without a trace — not even a SKIPPED row — both when the server lists no tools and when tools/list discovery fails outright. Both paths now surface them as untestable FAILUREs, from a single shared case table so the live and backfill paths can't drift.

SEP-2663 tasks scenarios: the four 'no task created' skips in tasks-lifecycle/tasks-wire-fields become untestable failures (the create-step failure was already red; the downstream IDs no longer read as silently green). The unconditional skip in tasks-status-notifications is intentionally untouched: that's a harness gap on a MAY requirement, not a server gap.

AGENTS.md documents the policy next to the severity convention: missing prerequisite → notTestable()/untestableCheck(); SKIPPED is reserved for legitimately-not-applicable checks.

Test-suite bug found en route: the stream mock in stateless.test.ts lacked releaseLock on its reader, so listenToStream's teardown threw, the helper swallowed the error, and every existing stream-based assertion was running against zero frames — passing on coincidentally-matching failure paths. Fixed; the existing subscription tests now exercise real frames (all still pass, now for the right reasons).

What deliberately stays SKIPPED

  • Capability-undeclared gates on optional features (prompts.listChanged / tools.listChanged SHOULD checks).
  • Runner-level spec-version inapplicability.
  • tasks-status-notifications (suite-side harness gap, MAY requirement).

How Has This Been Tested?

  • npm run check, npm run build, npm test — all green (328 tests / 31 files). all-scenarios.test.ts runs the draft suite against the everything-server, which implements all the diagnostic fixtures, so the reference run is unaffected by the stricter policy.
  • New tests pin both directions per flipped family: missing-tool → Not testable: FAILURE, listed-but-unenforced → genuine FAILURE, advertised-but-rejected listen → FAILURE, capability-never-declared listen → SKIPPED preserved; custom-header declared-ID untestable backfill (no SKIPPED row anywhere in that run); zero-tools and discovery-failure Mcp-Name rows (driven against a real local HTTP server, since that scenario sends its live cases over raw node http); tasks-lifecycle against a server that never creates a task; discover-never-observed listen rejection; untestableCheck shape.
  • Manual CLI run against a minimal fixture-less server produced the before/after outputs quoted above.

Breaking Changes

For SDKs whose servers lack the diagnostic fixtures: previously-green runs will now show failures naming the missing prerequisite. That is the intended behavior change — the affected scenarios can sit in expected-failures.yaml (the documented escape hatch) until the fixtures exist, and unlike before, building the fixture later flips the baseline entry to stale so the burn-down list stays honest. No check IDs changed; no aggregation/runner/schema changes.

Types of changes

  • Bug fix (non-breaking change which fixes an issue)
  • New feature (non-breaking change which adds functionality)
  • Breaking change (fix or feature that would cause existing functionality to change)
  • Documentation update

Checklist

  • I have read the MCP Documentation
  • My code follows the repository's style guidelines
  • New and existing tests pass locally
  • I have added appropriate error handling
  • I have added or updated documentation as needed

Additional context

  • This is the first slice of #248 (in-scenario prerequisite gates). Not covered here, as follow-ups: the client-direction all-SKIPPED degenerate case (http-standard-headers against a never-connecting client — interacts with the client-suite exit-code asymmetry in #245), the vacuous-SUCCESS predicates in input-required-result (A11/A15 pass when an unknown-tool error satisfies the predicate), the sse-retry/sse-polling early-return ID-set collapse, and a shared scenario-setup helper as #248 proposes.
  • Known limitations, called out in the messages: the diagnostic-tool detection reads a single page of tools/list (a server that paginates its diagnostic tools past the first page is classified by probe evidence instead — a result still proves execution), and the untestable WARNINGs on the SHOULD checks affect --expected-failures runs but not the bare server exit code, which ignores WARNING today (pre-existing asymmetry, #245).
  • details.untestable: true + the Not testable: prefix are the machine-readable markers; if we later want a dedicated UNSUPPORTED status in CheckStatus, these sites are already enumerable by grep.