test(examples): run each example file against the mocks
GL
13h ago+499+499+012345678901234567890123456789-367-367-012345678901234567890123456789
Supersedes #381 by @willleeney, rebased on current main and finished. Will's commit is kept as the first commit and credited as co-author on the rework.
Summary
- Each example test now imports the real file from
examples/and runs it, instead of re-implementing what the example does.tests/examples/run-example.tspoints the example at the mocks withSTACKONE_BASE_URL, capturesconsole.logandconsole.error, and turnsprocess.exitinto a thrown error so the vitest worker survives. - The OpenAI (Chat Completions and Responses) and Anthropic mocks run a real tool round: the model asks for
workday_list_workers, the example executes it against the MCP mock, and the model answers once the tool result comes back. vitest.setup.tsfails any request to a host other than127.0.0.1that no handler serves. Before this, MSW only warned, and running the Anthropic example from a test reached the realapi.anthropic.com(it got a 401 back with the dummy key).- Merging
maininto #381: the conflicts inmocks/handlers.tsand the claude-agent-sdk, openai and openai-responses tests resolve tomain. The tests for the defender-config and search-tools examples are dropped becausemaindeleted those examples. The claude-agent-sdk test stays as it is onmain, becausequery()there spawns a subprocess.
Test plan
- Each test asserts the example's exact output and its skip path (
Skipping: X_API_KEY is not set, exit 0), and checks which tools reached the MCP mock with which arguments. auth-managementserves a discovered account that hasworkday_list_workers, so section 5 runssetAccountId/getAccountIdinstead of its fallback. Its full output is pinned with an inline snapshot.- Mutation check: with each of the openai, ai-sdk and anthropic examples given a filter that drops the tool, the matching test fails.
- With the Anthropic handler removed, the test fails with MSW's unhandled-request error instead of reaching the network.
- Locally on the head commit (Node 24.22.3, pnpm 11):
pnpm lintis clean.pnpm testpasses 50 files, 1332 tests (4 skipped as onmain) with no type errors. The examples project passes 9 files and 18 tests.pnpm buildand publint are OK.
Review gates
| Gate | Result |
|---|---|
| Forge (loop and slop meter) | Comment lines 47 to 47, longest block 6, copied blocks 2 to 1 (the account-serving helper now lives in mocks/node.ts and the toolset tests use it). The one copy left is 3 lines of MCP mock wiring that also appear in src/toolsets.test.ts, outside this change. |
| Security review (Claude security reviewer, standing in for the Codex pass, which was unavailable) | No findings. No real credentials (only test-key style placeholders). The dynamic import takes a hardcoded example name at every call site. |
| Code review (Claude test reviewer) | Six findings. Fixed: the auth-management test only covered the fallback branch, the Anthropic mock ignored text blocks, two test titles claimed more than they asserted, and the stubExampleEnv doc was wrong. Left as they are: a second, event-based unhandled-request check (the setup guard already fails the test), and the single tool round in the mock (all the examples make one). |
| Aikido | Runs on this PR. |
Requested by guillaume@stackone.com in Slack (repo audit thread).
🤖 Generated with Claude Code
Summary by cubic
Example tests now run the real files from examples/ instead of re-implementing their logic, so a broken example fails CI instead of staying green.
tests/examples/run-example.tsimports each example, points it at the mocks, captures console output, and turnsprocess.exitinto an assertable exit code.- The OpenAI and Anthropic mocks run a real tool round: the model asks for
workday_list_workers, the example executes it against the MCP mock, and the model answers once the tool result is back. vitest.setup.tsfails any unmocked request to a remote host; previously the Anthropic test reached the realapi.anthropic.com.- Added the
workday_*tools to the MCP catalog, which the examples filter on but were missing, so the filters matched zero tools. - The claude-agent-sdk example is excluded because
query()spawns a subprocess MSW cannot intercept.