Make dbt PR reviewer demo-parity reliable

Merged
AN
5mo ago
+995-55

Summary

  • document the dbt PR reviewer self-improvement loop and acceptance matrix
  • thread dialect into core equivalence and fix --no-ai handling for deterministic demo runs
  • reduce safe-refactor noise and PII false positives while keeping real PII exposure blocking
  • add an opt-in DuckDB warehouse data-diff e2e and fix DuckDB connector open callback behavior under Bun

Core dependency

Acceptance evidence

  • Public demo matrix reached expected verdicts for all 6 branches:
    • demo/safe-refactor: APPROVE
    • demo/join-key-breakage: REQUEST_CHANGES
    • demo/test-removal: COMMENT
    • demo/new-pii-exposure: REQUEST_CHANGES
    • demo/mart-select-star: COMMENT
    • demo/incremental-without-guard: COMMENT
  • Real-world corpus floor preserved: 15/15 caught bad cases and 0/5 false positives.
  • DuckDB dbt e2e proof: demo/join-key-breakage had dbt build pass, while reviewer blocked pre-merge using deterministic altimate_core.structural_diff evidence.
  • Warehouse data-diff e2e: local DuckDB detected one head-only row and one updated value.

Focused validation

  • bun test --timeout 30000 test/altimate/review-runner.test.ts
  • bun test --timeout 30000 test/altimate/review.test.ts
  • bun run typecheck
  • bun run --conditions=browser script/review-realworld-eval.ts
  • ALTIMATE_RUN_WAREHOUSE_E2E=1 bun test --timeout 30000 test/altimate/data-diff-duckdb-e2e.test.ts
  • bun test --timeout 30000 test/altimate/data-diff-duckdb-e2e.test.ts
  • pre-push hook: bun turbo typecheck passed

Notes

  • The DuckDB data-diff test is skipped by default to keep the iteration loop fast; set ALTIMATE_RUN_WAREHOUSE_E2E=1 to run it.
  • Results and loop policy are tracked in docs/internal/2026-06-08-dbt-pr-review-self-improvement-loop.md.

Summary by cubic

Make the dbt PR reviewer deterministic and demo‑parity reliable with fixed PII recall, precise join‑risk gating, dialect‑aware equivalence, and sturdier DuckDB handling. All six public demo branches reach the expected verdicts, and the benchmark floor is preserved.

  • New Features

    • Dialect-aware SQL equivalence; thread manifest.metadata.adapter_type into altimate_core.equivalence (dialect now optional in the API).
    • Join-risk gating: map SC010 (join_key_regression) to join_risk; escalate to critical only when core reports severity: error (other structural rules stay advisory).
    • Diff‑scoped PII on newly introduced output columns with tuned confidence thresholds and exposure-aware escalation; runs concurrently; opt‑in DuckDB data‑diff e2e via @altimateai/drivers (ALTIMATE_RUN_WAREHOUSE_E2E=1).
  • Bug Fixes

    • PII recall: dedupe the broad schema.detect_pii fallback only against columns core actually classified; treat zero‑target lineage as uncertainty (keep fallback alive); make regex twin suppression column‑aware (pii_into_mart via its matched column, select-pii-columns only when a precise detector emitted a PII finding); keep deterministic fallback when lineage/classifier fails or misses.
    • Reduce safe‑refactor noise by suppressing regex/portability twins only when core emitted the same rule for the file.
    • DuckDB connector: arm the 2s fallback before replaying a synchronous open callback, close half‑open handles on error, retry READ_ONLY on lock, and propagate non‑lock errors.
    • CLI: respect --no-ai; programmatic ai: false also disables the advisory lane.

Summary by CodeRabbit

  • New Features

    • Added optional dialect parameter support for SQL equivalence analysis
    • Enhanced PII detection with improved diff-scoping and confidence-based classification
    • Improved join-risk detection in structural analysis
  • Bug Fixes

    • Strengthened DuckDB connection retry logic for locked database scenarios
    • Improved callback race-condition safety in database connection management
  • Tests

    • Added end-to-end test coverage for data diff operations with DuckDB
    • Expanded test coverage for equivalence analysis with dialect support