fix(examples): skip on missing credentials instead of passing vacuously

Closed
WI
2mo ago
+54-32

Summary

The example tests passed in CI while exercising nothing. This makes the result honest.

Every example guarded missing credentials with print(...); return. CI has no STACKONE_* or OPENAI_* secrets, and setup-nix runs uv sync --all-extras --locked, so all seven examples imported cleanly, printed a hint, returned, and passed.

Reproduced on main with no credentials set:

$ uv sync --all-extras
$ env -u STACKONE_API_KEY -u STACKONE_ACCOUNT_ID -u OPENAI_API_KEY uv run pytest examples
8 passed in 4.45s

That is worse than having no example tests, because it reads as coverage.

Changes

test_examples.py checks credentials before exec_module and skips. A skipped test is honest; a vacuously passing one is not. The two credential classes skip for different stated reasons:

ClassExamplesReason
StackOne onlyauth_management, search_toolsCan eventually run against a mock once STACKONE_BASE_URL is honoured (#197)
Live LLMopenai, langchain, langgraph, crewai, pydantic_aiNeeds OPENAI_API_KEY; cannot run against a mock StackOne server

The examples' own guards now raise rather than return. A guard that returns early is the mechanism that made the vacuous pass possible โ€” if the skip logic is ever weakened, raising fails loudly instead of silently going green. This also matches the pattern documented in .claude/rules/examples-standards.md.

openai_integration and crewai_integration additionally gained the OPENAI_API_KEY guard the other three LLM examples already had. Both construct an LLM client at runtime but neither checked for the key, so they would have raised from deep inside the client rather than at the top.

search_tools.search_and_execute keeps its print-and-return guard: it skips one section of an otherwise-working example, not the whole example.

On STACKONE_BASE_URL

No example passes base_url โ€” all construct StackOneToolSet() bare. So once #197 lands (SDK constructor reads STACKONE_BASE_URL), all seven honour it for free. No change was needed here, and none was made, to avoid duplicating that PR.

Verification

Before / after, same command, no credentials:

before:  8 passed in 4.45s
after:   1 passed, 7 skipped in 1.20s

Skip reasons are distinct per class โ€” with StackOne creds set but no OPENAI_API_KEY:

SKIPPED examples/test_examples.py:83: Skipping langchain_integration.py: needs a live LLM
        (OPENAI_API_KEY), so it cannot run against a mock StackOne server

Guards confirmed to raise rather than return silently:

auth_management:     raises ValueError(STACKONE_API_KEY is not set. Export it or add it to .env)
search_tools:        raises ValueError(STACKONE_API_KEY is not set. Export it or add it to .env)
openai_integration:  raises ValueError(STACKONE_API_KEY is not set. Export it or add it to .env)

Full suite with --all-extras: 298 passed, 27 skipped, 0 failures. ruff check clean, ruff format --check clean.

Note

These tests still cannot verify SDK behaviour in CI โ€” they now skip instead of lying about it. Actually exercising them requires pointing STACKONE_BASE_URL at a mock, which needs #197 plus a CI job that starts one. This PR makes the gap visible rather than hidden.

๐Ÿค– Generated with Claude Code


Summary by cubic

Fixes example tests passing vacuously in CI when credentials are missing โ€” they now skip honestly instead of reporting false coverage.

  • test_examples.py now skips tests when StackOne or OpenAI credentials are missing, with distinct skip reasons for the two credential classes.
  • Examples' credential guards now raise ValueError instead of printing and returning early.
  • openai_integration and crewai_integration gained the OPENAI_API_KEY guard the other LLM examples already had.
  • search_tools.search_and_execute keeps its print-and-return guard because it skips only one section of an otherwise-working example.