test(examples): execute the real example files

WI
2mo ago
+524-112

Summary

tests/examples/ reimplemented each example's logic against ../../src and never imported examples/*.ts. An example could rot or break outright and CI stayed green — and four of the eight had no test at all.

Seven of the eight examples now execute for real. Test count: 329 → 348.

How

Examples do their work at import time via top-level await, so importing one runs it. tests/examples/run-example.ts makes that safe under vitest:

  • STACKONE_BASE_URL points the SDK at the MSW mock (toolsets.ts:630 already reads it), so an example constructing new StackOneToolSet() with no explicit baseUrl hits the mock rather than production. No example file needed changing.
  • process.exit becomes a throw that runExample catches and reports as exitCode, so an example bailing on missing config is an assertable outcome rather than a torn-down vitest worker.

Coverage before / after

ExampleBeforeAfter
auth-management.tsno testexecuted (3)
search-tools.tsno testexecuted (3)
ai-sdk-integration.tsno testexecuted (2)
anthropic-integration.tsno testexecuted (2)
openai-integration.tsreimplemented (1)executed (2)
openai-responses-integration.tsreimplemented (1)executed (2)
defender-config.tsreimplemented (7)7 wire tests kept + 4 executed
claude-agent-sdk-integration.tssetup only (3)unchanged — see below

The one example not executed

claude-agent-sdk-integration.ts calls query(), which spawns a claude-code subprocess that MSW cannot intercept and which needs a real ANTHROPIC_API_KEY and installation. Its existing setup-level test is kept unchanged; the exclusion is now stated in the file rather than implied.

Mock gaps this uncovered

Each of these meant the examples exercised nothing:

  • The three workday_* actions every example filters on were absent from the MCP catalog, so the filters matched zero tools.
  • No Anthropic handler existed at all — added mocks/handlers.anthropic.ts.
  • The OpenAI handlers keyed off prompts none of the examples send; all four LLM examples prompt "List the first 5 employees".
  • Semantic search (/actions/search) was unmocked, so search-tools hit the real network (ECONNREFUSED). Added scoped to that test via server.use — a global handler pre-empts src/semantic-search.test.ts, which deliberately asserts against an unmocked endpoint. That cost me 6 failures before I scoped it.

Assertions are exact, and mutation-tested

Tool-count assertions are exact rather than > 0. A > 0 check cannot fail here: fetchTools appends tool_feedback after filtering, so a filter matching nothing still returns one tool.

I caught this by mutation-testing my own test — pointing an example at a nonexistent action initially passed. With exact counts it now fails correctly:

AssertionError: expected 1 to be 4

Orphan tests relabelled, not moved

fetch-tools.test.ts and tanstack-ai-integration.test.ts describe examples that do not exist. Relabelled in place rather than relocated: the examples vitest project globs only tests/examples/** and root globs only src/+scripts/, so moving them to tests/integration/ would have dropped them from both projects and silently stopped running them.

Test plan

  • pnpm test — 348 passed / 1 failed
  • pnpm run build — clean, publint no issues (the conformance harness at ../sdk-conformance imports dist/index.mjs; unaffected)
  • oxfmt --check . — clean across 107 files
  • knip — no unused findings

The single failure is pre-existing and environmental: scripts/package-exports.test.ts fails with spawn tsc ENOENT because tsc is not on PATH locally (it comes from nix). It fails identically on origin/main. oxfmt/oxlint are likewise not on PATH here, so formatting was verified via pnpm dlx oxfmt; CI's nix-pinned versions will run the real thing.

Noted, not fixed

  • fetchTools appends tool_feedback after the provider/action filters, so a filter matching nothing still yields one tool. Surfaced here as the reason exact counts are needed; not changed.
  • knip.config.ts has a pre-existing hint to remove @typescript/native-preview from ignoreDependencies.

🤖 Generated with Claude Code


Summary by cubic

Tests in tests/examples/ now execute the real examples/*.ts files instead of reimplementing each example's logic against ../../src. Seven of the eight examples now run for real (329 → 348 tests); four had no test at all before.

  • run-example.ts imports each example under MSW by pointing STACKONE_BASE_URL at the mock and turning process.exit into an assertable exit code.
  • Added the workday_* MCP actions, an Anthropic handler, OpenAI handlers for the examples' actual prompt, and a semantic-search mock — all previously missing, so the examples exercised nothing.
  • claude-agent-sdk-integration.ts stays setup-only because query() spawns a claude-code subprocess MSW can't intercept.
  • Tool-count assertions are exact, not > 0: fetchTools appends tool_feedback after filtering, so a > 0 check passes even when nothing matches.
  • The two tests for non-existent examples are relabelled in place; moving them would drop them from vitest's project globs.
  • The single failing test is pre-existing and environmental (spawn tsc ENOENT), failing identically on main.