feat(review): opt-in test-gap check and focused AI batches; verifier reads every changed file
Issue for this PR
Part of #1792
Type of change
- Bug fix
- New feature
- Refactor / code improvement
- Documentation
What does this PR do?
Stacked on #1771. Adds two opt-in AI-lane options, lets the verifier read every changed file, and fixes the rule gaps the planted-bug suite found. Defaults are unchanged.
What changed
ALTIMATE_REVIEW_AI_TEST_GAP=1: when a changed macro, materialization or generic test changes runtime behaviour, the AI says whether the PR adds a test that runs it, as onetest_coveragecomment. Each batch also gets the PR's other changed paths.ALTIMATE_REVIEW_AI_FOCUSED=1: on PRs with 7 or more files, related files are reviewed together in batches of up to 5. Related means a macro with its dispatch variants, a test with the model it refs, and a yml with the models it declares. Grounding is filtered per batch, findings are deduplicated, and there are at most 4 batches. No paid measurement yet, so it stays opt-in.- The verifier can read every file the PR changes, so it can confirm "no test was added".
- Rule fixes from the planted-bug suite:
- three recall rules;
- the join chain across a join with no ON clause;
- parse-breaking findings are kept in dev, sandbox and scratch folders.
Results
Test-gap check
- A private 5-project monorepo PR that changed a materialization with no test: the AI raised the missing test in 10 of 10 runs, against 2 of 21 before.
- 10 tuning-repo PRs: catches unchanged (5 of 16), useful comments 51% → 57%, wrong 3% → 0% (blind Claude judge), tokens 1.02×.
Planted-bug suite on repos never used for tuning: 236 of 238 caught, 1 false alarm on 237 no-defect twins.
How did you verify your code works?
bun test test/altimate/review: 748 pass.bunx tsc --noEmit: clean.- New:
test/altimate/review-ai-test-gap.test.ts, plus focused-batch cases inreview-ai-inputs.test.ts.
Cost side effects
None by default; both options are opt-in. ALTIMATE_REVIEW_AI_TEST_GAP=1 measured at about 1.02x AI-lane tokens. ALTIMATE_REVIEW_AI_FOCUSED=1 splits PRs with 7 or more files into at most 4 batches, so up to 4 AI calls per review instead of one (not yet measured, hence opt-in). Letting the verifier read every changed file only applies when the verifier is enabled and stays within its 12k-token cap. No warehouse queries or cloud resources.
Screenshots / recordings
Not a UI change.
Checklist
- I have tested my changes locally
- I have not included unrelated changes in this PR
- I checked for cost side effects (model calls, queries or cloud resources that grow with user data) and described them, or wrote "none"
🤖 Generated with Claude Code