review: --no-ai fix + --explain-tier / --force-tier flags + manifest auto-discovery + column-aware schema.yml test-removal

Merged
SA
4mo ago
+2004-56

Summary

Improvements to altimate-code review covering two bug fixes and four observability / recall features that share the same code surface. Filed against packages/opencode/src/{cli/cmd/review.ts, altimate/review/*.ts}.

Bug fixes

Both are reproducible against main and closed by this PR:

  • #1025 — bare --no-ai <other-flag> triggers the yargs help path and exits 0 without running the review. Fix: .parserConfiguration({ "boolean-negation": false }) on the review command's yargs builder, so --no-ai binds to the declared noAi option as authored. --no-ai=true / --no-ai=false continue to work for programmatic parity.
  • #1026 — detectSchemaYmlPatterns compared removed and added test lines as raw trimmed strings, so a genuine removal from one column was silently cancelled by an unrelated re-add on a sibling column (yaml re-serialization emits the same - unique / - not_null lines for other columns still declaring those tests). Fix: collectTestOccurrencesFromDiff in dbt-patterns.ts walks the unified diff with model/column context via nearest preceding - name: X header (resets on hunk boundaries), keys removals by (model, column, test), and only cancels on a same-tuple re-add. Emits one finding per removed (model, column, test) triple, with severity elevated to warning for unique removals and for not_null on id/key columns. Body text detects layer from file path (marts?/reporting → "mart-layer key", else "declared key") so the copy doesn't mislabel non-mart schema files.

Feature enhancements bundled with the fixes

Four additions in the same code surface, small enough to review alongside the bug fixes rather than as their own PRs:

  • --explain-tier — new boolean flag. When set, the verdict envelope carries tierReasons: string[] (the risk classifier's reasons for the tier decision) and the human-readable summary prints a 🧭 **Tier: X** — <reasons> line. Read-only — doesn't change tier classification or verdicts. Envelope field is optional (absent when the flag is off), so existing verdicts remain byte-equivalent.

  • --force-tier <trivial|lite|full> — new experimental flag for bench and debug work. Overrides the tier classifier with the supplied tier. Guardrails:

    • Prints an ⚠️ --force-tier=X is EXPERIMENTAL (bench / debug only). Classifier bypassed; verdict envelope will carry tierForced: true. warning to stderr on every use.
    • Verdict envelope carries tierForced: true and tierClassified: <original> whenever the flag is passed, regardless of whether the forced value matches the classifier's decision. (An earlier iteration of this change guarded that flag on input.forceTier !== classifiedTier, which meant --force-tier=full on a naturally-full PR silently bypassed the audit envelope. Corrected in the same branch — every use of the flag is now recorded.)
    • Verdict headline renders as <tier> tier — forced (was <classified>).
    • tierReasons is always populated when the flag is used, with a leading forced via --force-tier=X (classifier said Y) marker so downstream can't confuse the forced tier for a natural one.
    • Not a default-visible feature. Documented as EXPERIMENTAL in the yargs describe; the stderr banner ensures no one uses it in CI without noticing.
  • Manifest auto-discovery — before, --manifest <path> had to be explicit; otherwise the CLI used the config-relative default target/manifest.json, which silently missed when cwd wasn't the dbt project root, degrading every such review to lint-only. Now:

    1. If --manifest isn't explicit AND the config-relative path doesn't exist, walk UP from cwd looking for dbt_project.yml. Use the adjacent target/manifest.json when it exists.
    2. Log discovery to stderr (ℹ️ auto-discovered dbt manifest at ...) so users see which manifest the review used.
    3. Never auto-discover a manifest from a directory without a dbt project — a target/ from an unrelated tool never gets picked up.
    4. Freshness warning: when --head is set (CI shape) AND any changed file has an mtime newer than the manifest, print ⚠️ manifest ... appears stale to stderr. Skipped for working-tree diffs (mtime noise during active edits would spam warnings).

    Explicit --manifest <path> always wins. Auto-discovery is only attempted when the caller was silent AND the config default is absent.

  • Verdict envelope — three new optional fields added to VerdictEnvelope in verdict.ts, all included in the signed canonical body so tampering is detectable:

    • tierReasons?: string[] (from --explain-tier)
    • tierForced?: boolean (from --force-tier)
    • tierClassified?: RiskTier (from --force-tier — original tier before the override)

    All are optional and absent when the corresponding flag is off, so existing verdicts remain byte-equivalent to before.

Testing

Verified locally on synthetic scenarios:

  • Bare --no-ai now runs the review (verifies bug fix for #1025).
  • --no-ai=true and --no-ai=false continue to work.
  • --explain-tier populates tierReasons on both trivial and full verdicts.
  • --force-tier=full on a naturally-full PR sets tierForced: true and tierClassified: "full" on the envelope + emits the stderr warning + the leading forced via reason. --force-tier=lite on a naturally-full PR overrides tier to lite and records tierClassified: "full".
  • Manifest auto-discovery: finds target/manifest.json when invoked from a subdirectory containing dbt_project.yml upward; declines to use a target/manifest.json from a directory that has no dbt_project.yml (verified against a plain non-dbt git repo).
  • schema.yml test-removal: reproducer for #1026 (remove unique+not_null from one column while sibling columns still have them) now surfaces two warning-severity findings, one per removed (column, test) tuple. Layer-aware body renders "declared key" for a top-level models/schema.yml and "mart-layer key" for a models/marts/schema.yml.
  • Envelope signature still round-trips through verifyEnvelope for both flag-on and flag-off shapes.

Test plan

  • Bare --no-ai runs the review (repro from #1025 no longer occurs)
  • --no-ai=true / --no-ai=false behavior unchanged
  • --explain-tier produces tierReasons in JSON output; envelope signature verifies
  • --force-tier prints EXPERIMENTAL warning to stderr; envelope carries tierForced: true + tierClassified: <original> even when forced tier matches classifier
  • Manifest auto-discovery finds target/manifest.json from subdir; explicit --manifest still wins; auto-discovery declines outside a dbt project
  • Freshness warning fires when a changed file's mtime > manifest mtime AND --head is set; silent otherwise
  • schema.yml test-removal reproducer for #1026 now emits per-(column, test) findings
  • Existing verdicts (no new flags) are byte-equivalent to main

Closes #1025.
Closes #1026.


Summary by cubic

Fixes bare --no-ai runs, hardens file reads against symlink escapes with a shared safeReadInside helper, upgrades schema.yml test‑removal to a structural YAML diff (incl. deleted/renamed files), auto‑discovers the dbt manifest, and makes subdir/monorepo invocations resolve compiled SQL and working‑tree files correctly. Closes #1025 and #1026.

  • Bug Fixes

    • CLI: bare --no-ai now runs instead of falling into help.
    • Security: working‑tree and compiled‑SQL reads reject targets that realpath outside the repo/project (blocks tracked‑symlink escapes) and now share the centralized safeReadInside helper.
    • schema.yml: structural diff across models/sources/snapshots/seeds with tests/data_tests; handles deleted and renamed files; model‑level and column‑level attribution; per‑removal findings; elevated severity for unique and not_null on likely key columns; multiple relationships on one column are distinct; snapshot property files under snapshots/*.yml now route here; safe diff‑only fallback with per‑occurrence tags; emits the per‑file summary once with clearer “This schema file drops …” wording.
    • Tier display/audit: headline shows “forced (was X)” with an “unknown” fallback when tierClassified is missing; envelope schema enforces tierForced ↔ tierClassified consistency and rejects tierForced: false; backtick‑safe, capped tier reason rendering.
    • Stale‑manifest warning filters to dbt‑relevant paths and only treats .md under models/ or analyses/ as manifest‑affecting to avoid false positives.
    • Subdir/monorepo invocation: working‑tree reads root at the git repo top‑level; compiled SQL resolver uses the dbt project realpath and strips the repo→project path prefix (Windows‑ and symlink‑safe), so files like packages/dbt/models/... resolve under the discovered dbt project.
  • New Features

    • --explain-tier: attaches tierReasons[] to the envelope and prints a capped tier line in the summary.
    • --force-tier <trivial|lite|full> (experimental): overrides the classifier with a stderr warning; envelope records tierForced: true and tierClassified; reasons start with a “forced via …” marker; headline shows “forced (was X)”.
    • Manifest auto‑discovery: when --manifest is omitted and the config default is missing, walks up to the nearest dbt_project.yml and uses its target/manifest.json; logs the chosen path; never picks a manifest outside a dbt project; warns if stale when --head is set; routes compiled SQL resolution and project name to the discovered project root. Explicit --manifest always wins.
    • Verdict envelope: adds optional tierReasons, tierForced, and tierClassified (all signed). Backwards compatible when flags aren’t used.

Summary by CodeRabbit

  • New Features

    • Added --explain-tier to include tier reasoning in the verdict and enrich the summary when reasons are available.
    • Added experimental --force-tier to override tier selection and record forced-tier auditing metadata.
    • Auto-discovers the nearest dbt manifest when not provided, and warns if it may be stale.
  • Bug Fixes

    • Improved schema.yml guardrail removal detection with YAML-aware attribution (including data_tests), plus a safe diff-only fallback.
    • Fixed CLI boolean parsing for --no-ai.
    • Corrected snapshot classification under snapshots/ (now handles non-.sql snapshot YAML files).
  • Tests

    • Expanded coverage for tier rendering/forcing, schema-yml attribution, manifest staleness, and subdirectory invocation behavior.