test(examples): run each example file against the mocks

GL
13h ago
+499-367

Supersedes #381 by @willleeney, rebased on current main and finished. Will's commit is kept as the first commit and credited as co-author on the rework.

Summary

  • Each example test now imports the real file from examples/ and runs it, instead of re-implementing what the example does. tests/examples/run-example.ts points the example at the mocks with STACKONE_BASE_URL, captures console.log and console.error, and turns process.exit into a thrown error so the vitest worker survives.
  • The OpenAI (Chat Completions and Responses) and Anthropic mocks run a real tool round: the model asks for workday_list_workers, the example executes it against the MCP mock, and the model answers once the tool result comes back.
  • vitest.setup.ts fails any request to a host other than 127.0.0.1 that no handler serves. Before this, MSW only warned, and running the Anthropic example from a test reached the real api.anthropic.com (it got a 401 back with the dummy key).
  • Merging main into #381: the conflicts in mocks/handlers.ts and the claude-agent-sdk, openai and openai-responses tests resolve to main. The tests for the defender-config and search-tools examples are dropped because main deleted those examples. The claude-agent-sdk test stays as it is on main, because query() there spawns a subprocess.

Test plan

  • Each test asserts the example's exact output and its skip path (Skipping: X_API_KEY is not set, exit 0), and checks which tools reached the MCP mock with which arguments.
  • auth-management serves a discovered account that has workday_list_workers, so section 5 runs setAccountId / getAccountId instead of its fallback. Its full output is pinned with an inline snapshot.
  • Mutation check: with each of the openai, ai-sdk and anthropic examples given a filter that drops the tool, the matching test fails.
  • With the Anthropic handler removed, the test fails with MSW's unhandled-request error instead of reaching the network.
  • Locally on the head commit (Node 24.22.3, pnpm 11): pnpm lint is clean. pnpm test passes 50 files, 1332 tests (4 skipped as on main) with no type errors. The examples project passes 9 files and 18 tests. pnpm build and publint are OK.

Review gates

GateResult
Forge (loop and slop meter)Comment lines 47 to 47, longest block 6, copied blocks 2 to 1 (the account-serving helper now lives in mocks/node.ts and the toolset tests use it). The one copy left is 3 lines of MCP mock wiring that also appear in src/toolsets.test.ts, outside this change.
Security review (Claude security reviewer, standing in for the Codex pass, which was unavailable)No findings. No real credentials (only test-key style placeholders). The dynamic import takes a hardcoded example name at every call site.
Code review (Claude test reviewer)Six findings. Fixed: the auth-management test only covered the fallback branch, the Anthropic mock ignored text blocks, two test titles claimed more than they asserted, and the stubExampleEnv doc was wrong. Left as they are: a second, event-based unhandled-request check (the setup guard already fails the test), and the single tool round in the mock (all the examples make one).
AikidoRuns on this PR.

Requested by guillaume@stackone.com in Slack (repo audit thread).

🤖 Generated with Claude Code


Summary by cubic

Example tests now run the real files from examples/ instead of re-implementing their logic, so a broken example fails CI instead of staying green.

  • tests/examples/run-example.ts imports each example, points it at the mocks, captures console output, and turns process.exit into an assertable exit code.
  • The OpenAI and Anthropic mocks run a real tool round: the model asks for workday_list_workers, the example executes it against the MCP mock, and the model answers once the tool result is back.
  • vitest.setup.ts fails any unmocked request to a remote host; previously the Anthropic test reached the real api.anthropic.com.
  • Added the workday_* tools to the MCP catalog, which the examples filter on but were missing, so the filters matched zero tools.
  • The claude-agent-sdk example is excluded because query() spawns a subprocess MSW cannot intercept.