Slim Basecamp skill and rely on live CLI help

Merged
RO
13d ago
+354-1562

Intent

Keep the Basecamp skill focused on durable operating and safety rules instead of duplicating the CLI's command manual.

The existing skill had grown to 89 KB / 1,565 lines. Loading it for a simple question such as “how many Basecamp accounts do I have?” added roughly 13,000 tokens of tool output before the command could run, even though the installed CLI already exposes structured, command-specific --agent --help.

What changed

  • reduce the main skill to 8.3 KB / 235 lines (about 91% smaller)
  • make live basecamp ... --agent --help the source of truth for command syntax, flags, scope, and notes
  • retain cross-cutting rules for credentials, URLs and replies, output modes, project/account scope, stdin content, attachments, retries, authentication, and mutations
  • teach the eval harness to serve structured help from the locally built CLI
  • add an eval for the original account-count question
  • enforce a 16 KiB skill budget and discovery-first contract in Go tests
  • document deterministic checks and model-eval comparison workflows

Why the eval harness changed

A discovery-first skill cannot be evaluated faithfully if help calls receive the generic empty mock. The harness now executes only structured --agent --help requests against the local binary; normal Basecamp operations remain deterministic mocks.

Verification

  • GOWORK=off bin/ci
  • eval patterns: 162 compiled across 30 cases
  • structured-help harness self-test
  • skill drift checks

The hosted model eval was not run locally because ANTHROPIC_API_KEY is not configured; the existing PR eval workflow can run it when the repository secret is available.

Summary by cubic

Slims the Basecamp skill from 89 KB / 1,565 lines to 8.3 KB / 235 lines by removing the embedded command manual and treating live basecamp ... --agent --help output as the source of truth for syntax, flags, and scope. Loading the old skill added roughly 13,000 tokens for simple questions; the new one keeps only durable operating and safety rules like credentials, output modes, and mutation retries. Discovery now goes leaf-first: accounts list --agent --help directly instead of walking down the command tree.

Eval harness

  • Serves structured --agent --help from the locally built CLI and enforces leaf-first discovery; other Basecamp operations remain deterministic mocks.
  • Adds eval cases for the account-count question and Docs & Files folder downloads, with tightened mock and matcher regexes that enforce correct command and flag sequences, plus a self-test for the structured help path, runnable via make check-eval-harness.
  • Go tests enforce a 16 KiB skill budget and the discovery-first contract.
  • Clarifies that the todolist --description flag accepts rich text HTML, with a test guarding the updated help text.