feat(review): post fewer, better-placed comments: rule tiers, inline budget, pre-existing tags, coverage ledger
RA
11h ago+2941+2941+0123456789012345678901234567890123456789-296-296-012345678901234567890123456789
Stacked on #1739. This PR changes what the review posts and where, so each PR gets fewer, better-placed comments. It adds no model calls.
Results (replay of public dbt PRs, rules only)
| Split | Findings per PR, before → after | Reviewer-found defects caught, before → after |
|---|---|---|
| Tuning repos, 342 PRs | 15.6 → 10.0 | 52 → 51 |
| Repos never used for tuning, 71 PRs | 8.0 → 5.1 | 10 → 10 |
- At most 10 inline comments per PR. 99–100% of them sit on a line the PR changed (before: 35–75%).
- Inline rule comments judged blind: 48% useful and 5% wrong (before: 7% useful, 29% wrong).
- Planted-bug suite (104 injected defects, each with a no-defect twin): 100 caught, 0 false alarms on the twins.
What changed
Posting policy
- Every finding carries a stable
ruleid. - A per-rule tier table (
rule-tiers.ts) puts each rule inline, in the summary, or off. Repos can override it withrules:in the review config, which is read from the base branch. Engine checks that decide the verdict can only be turned off by an exact key. - Rules whose tier changed when one repo was left out of the judging sample stay at summary until they are re-judged.
- Findings on lines the PR did not add are tagged pre-existing. They never post inline and never affect the verdict.
postingconfig: an inline budget (default 10), a severity floor (defaultwarning), and a silent clean run (default on).
Placement and form
- Rule findings are anchored to the diff line they matched, including Jinja models (by compiled line text) and removed columns.
- Bodies are capped at 600 characters when rendered (the JSON envelope keeps the full text). The matched line is quoted, and valid fixes post as GitHub suggestion blocks.
- The summary lists inline comments, then the remaining findings by severity, then pre-existing ones.
Coverage
- A ledger lists every changed file the review did not look at, with the reason (excluded, out of scope, no checks for that kind, no manifest, undecidable equivalence).
macro-pathsthat point outside the dbt project root are honoured.
Measurement
- The AI lane records tokens, cost and calls in the envelope and telemetry.
- Opt-in switches let the AI lane see whole files and dbt parents (
ALTIMATE_REVIEW_AI_MAX_FILES,ALTIMATE_REVIEW_AI_MAX_FILE_CHARS,ALTIMATE_REVIEW_AI_BATCH_CHARS,ALTIMATE_REVIEW_AI_GRAPH). All are off by default.
Rule fixes from the planted-bug suite
case-duplicate-when,IGNORE NULLSplacement,temporary-code-markerphrasing,microbatch-granularity-mismatchadapter gating,select *through CTEs.removed-final-layer-columnsis raised to warning so it can post inline.
Testing
bun test test/altimate/review: 591 pass.bunx tsc --noEmit: clean.
🤖 Generated with Claude Code
Summary by cubic
Changes what the dbt review posts and where, so each PR gets fewer, better-placed comments without any new model calls. Replayed on public dbt PRs, findings per PR drop from 15.6 to 10.0 (8.0 to 5.1 on repos never used for tuning) while keeping the same reviewer-detected defects, and blind judging rates 48% of inline comments useful (up from 7%) and 5% wrong (down from 29%).
Posting policy
- Every finding carries a stable
ruleid; a new default tier table puts each rule inline, in the summary, or off. Repos override it withrules:in the review config, read from the base branch so a PR cannot relax its own review, and verdict-deciding engine checks are only disabled by exact key. - Findings on lines the PR did not add are tagged pre-existing: they never post inline and never count toward the verdict. The
postingconfig adds an inline budget (10), a severity floor (warning), and silent clean runs that post nothing. - Rule findings anchor to the matched diff line, including Jinja models (by compiled line text) and removed columns; rendered bodies cap at 600 characters, the matched line is quoted, and valid fixes post as GitHub suggestion blocks.
- The summary lists inline comments, then remaining findings by severity, then pre-existing ones. Rules whose tier depended on the judging sample stay at summary until re-judged.
Coverage and measurement
- A new ledger lists every changed file the review did not look at, with the reason (excluded, out of scope, no matching checks, no manifest, undecidable equivalence), and
macro-pathsoutside the project root are honored. - The AI lane records tokens, cost, and calls in the envelope and telemetry; opt-in switches let it see whole files and dbt parents, all off by default.
- Planted-bug fixes:
case-duplicate-whenspacing,IGNORE NULLSplacement,temporary-code-markerphrasing,microbatch-granularity-mismatchadapter gating, andselect *lineage through CTEs.removed-final-layer-columnsis raised to warning so it can post inline. 591 tests pass andtsc --noEmitis clean.