feat(review): telemetry for review feature usage

Merged
SA
4mo ago
+862-46

Issue for this PR

Closes #1065

Type of change

  • Bug fix
  • New feature
  • Refactor / code improvement
  • Documentation

What does this PR do?

The review feature emitted no telemetry at all — no Telemetry reference in cli/cmd/review.ts or the engine, and no session on the CLI path, so not even session_start. We couldn't tell how often review runs, who runs it, what it finds, or whether results get published.

Adds two events from one helper shared by both callers of the engine — the review command and the dbt_pr_review tool — separated by an invocation field. Instrumenting only the command would have missed every review an agent runs through the tool.

Caller attribution needed no code. toAppInsightsEnvelopes already seeds source from Flag.ALTIMATE_CLI_CLIENT, and an event only overrides that by declaring its own source field. These events declare none, so anything exporting ALTIMATE_CLI_CLIENT is attributed for free. That let me drop an earlier design carrying a global envelope property, a per-command entrypoint registry, a reviewPullRequest() signature change and typed errors through the engine.

Four choices that aren't obvious from the diff:

  • Publication is a separate event. It happens after the review is computed and can partially succeed, so it can't be a field on review_run, and a failed post must not mark a computed review as failed.
  • partial covers every degraded post state. PostResult can't distinguish them — postError isn't cleared when the retry succeeds, and an inline fallback coexists with a real reviewId.
  • Two failure reasons plus a fallback. The engine degrades rather than throwing for missing manifests, dispatcher failures and the AI lane, so buckets for those would never fire.
  • by_category is zero-filled from the 14-value enum and drops unknown keys, so "never fired" differs from "not possible" and a malformed category can't mint a new dimension.

Enums, booleans and counts only — review findings are about customer schema.

How did you verify your code works?

  • 13 unit tests. Mutation-checked rather than assumed: adding a source field to the event fails the attribution test, and removing the zero-fill fails two others.
  • One opt-in e2e (ALTIMATE_E2E=1 bun test test/e2e/, ~13s) that runs a real altimate-code review process against a real git repo, with telemetry pointed at a local HTTP sink. Spawned with only ALTIMATE_CLI_CLIENT set, the arriving envelope carried source=plugin:claude-code — which is the one claim the whole design rests on and can't be shown in-process.
  • That run also caught something I'd documented wrongly: a repo with a dbt_project.yml but no compiled manifest reports degraded=true, so degraded is a fidelity flag, not a "no warehouse" flag. Docs corrected.
  • Privacy asserted, not assumed: a unit test checks no schema identifier appears in the serialized event, and the e2e greps the whole payload for the fixture's file name, changed SQL and repo path.
  • tsgo --noEmit clean on packages/opencode; 162 telemetry tests passing.

Screenshots / recordings

N/A — no UI change.

Checklist

  • I have tested my changes locally
  • I have not included unrelated changes in this PR

🤖 Generated with Claude Code

https://claude.ai/code/session_018SLUQF3xgZHsGZHSjxe7vb

Summary by CodeRabbit

  • New Features

    • Added telemetry for review execution and publication outcomes across CLI and tool-based reviews.
    • Captures completion, failure, duration, verdict, quality metrics, and posting results while protecting repository and customer content.
    • Preserves existing review output and error behavior when telemetry is unavailable.
  • Documentation

    • Documented review telemetry events, payload fields, privacy safeguards, publication statuses, and event correlation.