fix(cli): stop error-tracking expected user errors and per-version fingerprints

Merged
PO
2mo ago
+78-9

Summary

Ordinary user mistakes from the CLI (bad/missing API key, app not created yet) were being reported to error tracking and re-fingerprinted on every release, so the same errors kept spawning brand-new issues instead of one triageable issue.

Why: real signal was shredded into noise — the bundle upload org-lookup failure alone fired ~1.4K times over 90 days, split across 40+ per-version fingerprints, and every CLI release triggered a fresh round of "new issue" alerts for problems that have existed for months. Users hit the same path got an opaque message with no hint about what to do.

Two root causes, both in the CLI:

  • Version in the fingerprint — capturePosthogException folded cli:<version>:<command> into $exception_fingerprint, so every release re-fingerprinted identical errors into a fresh issue.
  • Everything captured — the top-level handler sent all non-Commander throws to error tracking, including plain user-configuration failures.

Plus, getOrganizationId discarded its formatError(error) result (losing the real PostgREST cause), and bundle upload ran the org lookup before the app-existence/permission check — so a missing app failed with Cannot get organization id instead of the actionable "run app add" message.

Changes

AreaBeforeAfter
posthog.ts fingerprintincludes CLI version → new issue per releaseversion-less (cli:<command>:…); version still sent as cli_version
index.ts top-level handlercaptures all non-Commander errorsskips capture for expected user errors (401 invalid_apikey/no_key_provided, app-not-found); still calls trackCommandFailed
getOrganizationIdformatError(error) result discardedunderlying cause logged alongside the message
bundle upload orderorg lookup → (much later) app/permission checkapp existence + upload permission checked before org lookup

Expected user errors are identified by a new isExpectedUserError helper (message markers + 401 status, including supabase-js FunctionsHttpError.context.status). Genuinely unexpected failures — including a real getOrganizationId throw for an app that does exist — are still captured.

Test plan

  • bun test/test-posthog-exception.mjs — extended to assert the fingerprint no longer contains the version and that isExpectedUserError classifies the invalid-key / no-key / app-not-found / 401 cases as expected and other errors as not.
  • bun run lint and bun run typecheck pass.

Screenshots

N/A — telemetry/error-classification change, no visible CLI output change beyond the org-lookup error now including its cause.

Checklist

  • My code follows the code style of this project and passes bun run lint.
  • My change requires a change to the documentation.
  • I have updated the documentation accordingly.
  • My change has adequate E2E test coverage.
  • I have tested my code manually, and I have provided steps how to reproduce my tests.

Created with PostHog Desktop from this inbox report.

View with [code]smith Autofix with [code]smith
Need help on this PR? Tag @codesmith-bot with what you need. Autofix is disabled.