feat(review): post fewer, better-placed comments: rule tiers, inline budget, pre-existing tags, coverage ledger

RA
11h ago
+2941-296

Stacked on #1739. This PR changes what the review posts and where, so each PR gets fewer, better-placed comments. It adds no model calls.

Results (replay of public dbt PRs, rules only)

SplitFindings per PR, before → afterReviewer-found defects caught, before → after
Tuning repos, 342 PRs15.6 → 10.052 → 51
Repos never used for tuning, 71 PRs8.0 → 5.110 → 10
  • At most 10 inline comments per PR. 99–100% of them sit on a line the PR changed (before: 35–75%).
  • Inline rule comments judged blind: 48% useful and 5% wrong (before: 7% useful, 29% wrong).
  • Planted-bug suite (104 injected defects, each with a no-defect twin): 100 caught, 0 false alarms on the twins.

What changed

Posting policy

  • Every finding carries a stable rule id.
  • A per-rule tier table (rule-tiers.ts) puts each rule inline, in the summary, or off. Repos can override it with rules: in the review config, which is read from the base branch. Engine checks that decide the verdict can only be turned off by an exact key.
  • Rules whose tier changed when one repo was left out of the judging sample stay at summary until they are re-judged.
  • Findings on lines the PR did not add are tagged pre-existing. They never post inline and never affect the verdict.
  • posting config: an inline budget (default 10), a severity floor (default warning), and a silent clean run (default on).

Placement and form

  • Rule findings are anchored to the diff line they matched, including Jinja models (by compiled line text) and removed columns.
  • Bodies are capped at 600 characters when rendered (the JSON envelope keeps the full text). The matched line is quoted, and valid fixes post as GitHub suggestion blocks.
  • The summary lists inline comments, then the remaining findings by severity, then pre-existing ones.

Coverage

  • A ledger lists every changed file the review did not look at, with the reason (excluded, out of scope, no checks for that kind, no manifest, undecidable equivalence).
  • macro-paths that point outside the dbt project root are honoured.

Measurement

  • The AI lane records tokens, cost and calls in the envelope and telemetry.
  • Opt-in switches let the AI lane see whole files and dbt parents (ALTIMATE_REVIEW_AI_MAX_FILES, ALTIMATE_REVIEW_AI_MAX_FILE_CHARS, ALTIMATE_REVIEW_AI_BATCH_CHARS, ALTIMATE_REVIEW_AI_GRAPH). All are off by default.

Rule fixes from the planted-bug suite

  • case-duplicate-when, IGNORE NULLS placement, temporary-code-marker phrasing, microbatch-granularity-mismatch adapter gating, select * through CTEs.
  • removed-final-layer-columns is raised to warning so it can post inline.

Testing

  • bun test test/altimate/review: 591 pass.
  • bunx tsc --noEmit: clean.

🤖 Generated with Claude Code


Summary by cubic

Changes what the dbt review posts and where, so each PR gets fewer, better-placed comments without any new model calls. Replayed on public dbt PRs, findings per PR drop from 15.6 to 10.0 (8.0 to 5.1 on repos never used for tuning) while keeping the same reviewer-detected defects, and blind judging rates 48% of inline comments useful (up from 7%) and 5% wrong (down from 29%).

Posting policy

  • Every finding carries a stable rule id; a new default tier table puts each rule inline, in the summary, or off. Repos override it with rules: in the review config, read from the base branch so a PR cannot relax its own review, and verdict-deciding engine checks are only disabled by exact key.
  • Findings on lines the PR did not add are tagged pre-existing: they never post inline and never count toward the verdict. The posting config adds an inline budget (10), a severity floor (warning), and silent clean runs that post nothing.
  • Rule findings anchor to the matched diff line, including Jinja models (by compiled line text) and removed columns; rendered bodies cap at 600 characters, the matched line is quoted, and valid fixes post as GitHub suggestion blocks.
  • The summary lists inline comments, then remaining findings by severity, then pre-existing ones. Rules whose tier depended on the judging sample stay at summary until re-judged.

Coverage and measurement

  • A new ledger lists every changed file the review did not look at, with the reason (excluded, out of scope, no matching checks, no manifest, undecidable equivalence), and macro-paths outside the project root are honored.
  • The AI lane records tokens, cost, and calls in the envelope and telemetry; opt-in switches let it see whole files and dbt parents, all off by default.
  • Planted-bug fixes: case-duplicate-when spacing, IGNORE NULLS placement, temporary-code-marker phrasing, microbatch-granularity-mismatch adapter gating, and select * lineage through CTEs. removed-final-layer-columns is raised to warning so it can post inline. 591 tests pass and tsc --noEmit is clean.