Make dbt PR reviewer demo-parity reliable
Merged
AN
5mo ago+995+995+012345678901234567890123456789-55-55-01234567890123456789
Summary
- document the dbt PR reviewer self-improvement loop and acceptance matrix
- thread dialect into core equivalence and fix
--no-aihandling for deterministic demo runs - reduce safe-refactor noise and PII false positives while keeping real PII exposure blocking
- add an opt-in DuckDB warehouse data-diff e2e and fix DuckDB connector open callback behavior under Bun
Core dependency
- Depends on deterministic core improvements in AltimateAI/altimate-core-internal#140.
- Local validation used a locally built/linked
@altimateai/altimate-corefrom that branch.
Acceptance evidence
- Public demo matrix reached expected verdicts for all 6 branches:
demo/safe-refactor:APPROVEdemo/join-key-breakage:REQUEST_CHANGESdemo/test-removal:COMMENTdemo/new-pii-exposure:REQUEST_CHANGESdemo/mart-select-star:COMMENTdemo/incremental-without-guard:COMMENT
- Real-world corpus floor preserved:
15/15caught bad cases and0/5false positives. - DuckDB dbt e2e proof:
demo/join-key-breakagehaddbt buildpass, while reviewer blocked pre-merge using deterministicaltimate_core.structural_diffevidence. - Warehouse data-diff e2e: local DuckDB detected one head-only row and one updated value.
Focused validation
bun test --timeout 30000 test/altimate/review-runner.test.tsbun test --timeout 30000 test/altimate/review.test.tsbun run typecheckbun run --conditions=browser script/review-realworld-eval.tsALTIMATE_RUN_WAREHOUSE_E2E=1 bun test --timeout 30000 test/altimate/data-diff-duckdb-e2e.test.tsbun test --timeout 30000 test/altimate/data-diff-duckdb-e2e.test.ts- pre-push hook:
bun turbo typecheckpassed
Notes
- The DuckDB data-diff test is skipped by default to keep the iteration loop fast; set
ALTIMATE_RUN_WAREHOUSE_E2E=1to run it. - Results and loop policy are tracked in
docs/internal/2026-06-08-dbt-pr-review-self-improvement-loop.md.
Summary by cubic
Make the dbt PR reviewer deterministic and demo‑parity reliable with fixed PII recall, precise join‑risk gating, dialect‑aware equivalence, and sturdier DuckDB handling. All six public demo branches reach the expected verdicts, and the benchmark floor is preserved.
-
New Features
- Dialect-aware SQL equivalence; thread
manifest.metadata.adapter_typeintoaltimate_core.equivalence(dialect now optional in the API). - Join-risk gating: map SC010 (
join_key_regression) tojoin_risk; escalate to critical only when core reportsseverity: error(other structural rules stay advisory). - Diff‑scoped PII on newly introduced output columns with tuned confidence thresholds and exposure-aware escalation; runs concurrently; opt‑in DuckDB data‑diff e2e via
@altimateai/drivers(ALTIMATE_RUN_WAREHOUSE_E2E=1).
- Dialect-aware SQL equivalence; thread
-
Bug Fixes
- PII recall: dedupe the broad
schema.detect_piifallback only against columns core actually classified; treat zero‑target lineage as uncertainty (keep fallback alive); make regex twin suppression column‑aware (pii_into_martvia its matched column,select-pii-columnsonly when a precise detector emitted a PII finding); keep deterministic fallback when lineage/classifier fails or misses. - Reduce safe‑refactor noise by suppressing regex/portability twins only when core emitted the same rule for the file.
- DuckDB connector: arm the 2s fallback before replaying a synchronous open callback, close half‑open handles on error, retry READ_ONLY on lock, and propagate non‑lock errors.
- CLI: respect
--no-ai; programmaticai: falsealso disables the advisory lane.
- PII recall: dedupe the broad
Summary by CodeRabbit
-
New Features
- Added optional dialect parameter support for SQL equivalence analysis
- Enhanced PII detection with improved diff-scoping and confidence-based classification
- Improved join-risk detection in structural analysis
-
Bug Fixes
- Strengthened DuckDB connection retry logic for locked database scenarios
- Improved callback race-condition safety in database connection management
-
Tests
- Added end-to-end test coverage for data diff operations with DuckDB
- Expanded test coverage for equivalence analysis with dialect support