fix: fail checks whose prerequisite is missing instead of skipping them
A check whose prerequisite is missing — the server under test lacks a diagnostic fixture tool, or advertises a feature it then rejects — now fails with a Not testable: cause instead of reporting SKIPPED. SKIPPED is excluded from pass/fail counts, exit codes, and the expected-failures baseline, so these gaps read as green: a server could dodge the entire SEP-2575 capability-constraint and subscription surface (7 MUST-level checks) and still exit 0. This implements the policy direction from #248: a scenario that can't execute counts as a failure; the escape hatches are scenario filtering and the expected-failures baseline, not in-scenario skipping.
Motivation and Context
Concretely, before this change, a draft server with no diagnostic tools that advertises tools.listChanged but rejects subscriptions/listen:
sep-2575-server-rejects-undeclared-capability SKIPPED
sep-2575-missing-capability-http-400 SKIPPED
sep-2575-http-server-no-independent-requests… SKIPPED
sep-2575-server-no-log-without-loglevel SKIPPED
sep-2575-server-sends-subscription-ack SKIPPED
sep-2575-server-tags-subscription-id SKIPPED
sep-2575-server-honors-notification-filter SKIPPED
Passed: 10/21, 11 failed
The seven SKIPPED MUSTs vanish from the denominator, don't affect the exit code, and evaluateBaseline treats them as passing — worse, a baselined failing scenario whose checks degrade to SKIPPED (the server drops a fixture tool) flips its baseline entry to "stale — remove", which reads as fixed. After:
Passed: 10/28, 18 failed, 1 warnings
Error: Not testable: server does not list the diagnostic tool 'test_missing_capability' in tools/list and the probe call did not exercise it (required for the undeclared-capability rejection)
Error: Not testable: server advertises listChanged/subscribe capabilities but answers subscriptions/listen with -32601 (Method not found)
…
Every unexercised requirement is red with its cause named; the one remaining SKIPPED is the legitimately-inapplicable case (prompts.listChanged never declared).
What changed
New src/scenarios/untestable.ts — notTestable(reason) formats the stable Not testable: errorMessage prefix and untestableCheck(...) builds a failing check with details.untestable: true, so reports and dashboards can distinguish "violated" from "could not be verified" without a new status enum. Severity follows the underlying requirement's keyword (MUST → FAILURE, SHOULD → WARNING).
server-stateless:
- The capability-constraint pair now distinguishes a fixture gap from a genuine violation, strongest evidence first: a
resultfrom the probe call proves the tool executed → genuine MUST failure regardless of whethertools/listmentioned it; a non--32021error from a listed tool → wrong-code failure; an error from an unlisted tool → untestable failure naming the tool. Thetools/listsnapshot is fetched once and reused by the capabilities-match check (which now reports its own probe failure instead of misattributing it). - The streaming/logging probes (
test_streaming_elicitation,test_logging_tool) fail with the missing tool named instead of skipping. - The three subscription checks are now capability-aware:
-32601onsubscriptions/listenis a legitimate SKIP only when an observedserver/discoveradvertisement shows nolistChanged/subscribecapability. A server that advertises them but rejects the method fails (it claims a feature it does not serve), and when discover itself was never observed the gap is untestable rather than silently excused. - The trigger-hook gates on the two list-changed SHOULD checks report untestable WARNINGs (severity follows the keyword) instead of skipping. The capability-undeclared gates on those same checks stay SKIPPED — that's the genuinely optional case.
- The
sep-2575-http-server-error-jsonrpc-idSUCCESS backfill is now conditional on at least one error response having been observed (counted at thesendRpctransport seam, so every probe contributes); a server that wrongly accepted every invalid probe no longer collects a vacuous pass for the id-echo requirement.
SEP-2243 server scenarios:
http-custom-header-server-validation: the no-x-mcp-header-tool and no-string-param gates emit untestable FAILUREs (gate row + all four declared requirement IDs) instead of an all-SKIPPED run that previously passed with zero coverage of four MUSTs. The declared-ID backfill keeps the emitted ID set stable for traceability.http-header-validation: the three Mcp-Name cases (mismatch-reject, missing-reject, whitespace-accept) were previously omitted without a trace — not even a SKIPPED row — both when the server lists no tools and whentools/listdiscovery fails outright. Both paths now surface them as untestable FAILUREs, from a single shared case table so the live and backfill paths can't drift.
SEP-2663 tasks scenarios: the four 'no task created' skips in tasks-lifecycle/tasks-wire-fields become untestable failures (the create-step failure was already red; the downstream IDs no longer read as silently green). The unconditional skip in tasks-status-notifications is intentionally untouched: that's a harness gap on a MAY requirement, not a server gap.
AGENTS.md documents the policy next to the severity convention: missing prerequisite → notTestable()/untestableCheck(); SKIPPED is reserved for legitimately-not-applicable checks.
Test-suite bug found en route: the stream mock in stateless.test.ts lacked releaseLock on its reader, so listenToStream's teardown threw, the helper swallowed the error, and every existing stream-based assertion was running against zero frames — passing on coincidentally-matching failure paths. Fixed; the existing subscription tests now exercise real frames (all still pass, now for the right reasons).
What deliberately stays SKIPPED
- Capability-undeclared gates on optional features (
prompts.listChanged/tools.listChangedSHOULD checks). - Runner-level spec-version inapplicability.
tasks-status-notifications(suite-side harness gap, MAY requirement).
How Has This Been Tested?
npm run check,npm run build,npm test— all green (328 tests / 31 files).all-scenarios.test.tsruns the draft suite against the everything-server, which implements all the diagnostic fixtures, so the reference run is unaffected by the stricter policy.- New tests pin both directions per flipped family: missing-tool →
Not testable:FAILURE, listed-but-unenforced → genuine FAILURE, advertised-but-rejected listen → FAILURE, capability-never-declared listen → SKIPPED preserved; custom-header declared-ID untestable backfill (no SKIPPED row anywhere in that run); zero-tools and discovery-failure Mcp-Name rows (driven against a real local HTTP server, since that scenario sends its live cases over raw node http); tasks-lifecycle against a server that never creates a task; discover-never-observed listen rejection;untestableCheckshape. - Manual CLI run against a minimal fixture-less server produced the before/after outputs quoted above.
Breaking Changes
For SDKs whose servers lack the diagnostic fixtures: previously-green runs will now show failures naming the missing prerequisite. That is the intended behavior change — the affected scenarios can sit in expected-failures.yaml (the documented escape hatch) until the fixtures exist, and unlike before, building the fixture later flips the baseline entry to stale so the burn-down list stays honest. No check IDs changed; no aggregation/runner/schema changes.
Types of changes
- Bug fix (non-breaking change which fixes an issue)
- New feature (non-breaking change which adds functionality)
- Breaking change (fix or feature that would cause existing functionality to change)
- Documentation update
Checklist
- I have read the MCP Documentation
- My code follows the repository's style guidelines
- New and existing tests pass locally
- I have added appropriate error handling
- I have added or updated documentation as needed
Additional context
- This is the first slice of #248 (in-scenario prerequisite gates). Not covered here, as follow-ups: the client-direction all-SKIPPED degenerate case (
http-standard-headersagainst a never-connecting client — interacts with the client-suite exit-code asymmetry in #245), the vacuous-SUCCESS predicates ininput-required-result(A11/A15 pass when an unknown-tool error satisfies the predicate), the sse-retry/sse-polling early-return ID-set collapse, and a shared scenario-setup helper as #248 proposes. - Known limitations, called out in the messages: the diagnostic-tool detection reads a single page of
tools/list(a server that paginates its diagnostic tools past the first page is classified by probe evidence instead — aresultstill proves execution), and the untestable WARNINGs on the SHOULD checks affect--expected-failuresruns but not the bareserverexit code, which ignores WARNING today (pre-existing asymmetry, #245). details.untestable: true+ theNot testable:prefix are the machine-readable markers; if we later want a dedicatedUNSUPPORTEDstatus inCheckStatus, these sites are already enumerable by grep.