fix(review): verdict trust: only the engine blocks, policy from the base branch, allowForceTier

RA
11h ago
+420-30

Issue for this PR

Closes #

Stacked on #1721, which is stacked on #1720 and #1719; merge in that order.

Type of change

  • Bug fix
  • New feature
  • Refactor / code improvement
  • Documentation

What does this PR do?

The README and the docs promise "only the deterministic engine blocks". In the code, any three non-AI warnings, or one regex-catalog critical (join-no-on, create-or-replace, pii_into_mart, ā€¦), produced REQUEST_CHANGES. Separately, .altimate/review.yml was read from the PR's own checkout. So a PR could set mode: comment, drop a blockOn category, exclude its own paths, or narrow reviewers, and pass under its own relaxed rules.

  • Who can block: findings now carry origin: engine | catalog | ai, derived from evidence.tool. Unknown tools count as catalog, so a new detector can't block until someone classifies it. Only engine findings count toward REQUEST_CHANGES. A pattern rule a team trusts can be allowed to block via rubric.blockingRules: [<rule id>]. origin is stamped in buildEnvelope and covered by the signature.
  • Policy source: .altimate/review.yml is read at the merge-base commit (git show <base>:<path>). The PR's version is parsed only to list what it changes; those settings show in the comment and in policy.changedInPr and apply once merged. The envelope signs policy.source and policy.hash. Callers without git (pre-supplied files and content) still read the working tree.
  • --force-tier: allowForceTier: false in the base policy rejects the flag. Forced runs were already recorded as tier_forced in review_run.

Behavior changes:

  • Pattern-rule criticals and warning counts no longer block by default.
  • A PR's edits to review.yml no longer apply to that PR.
  • The tests that asserted the old behavior are updated: the two pii_into_mart regex-twin tests now expect COMMENT, and the verdict unit tests use engine evidence.

Not covered here: CLI flags in the workflow file (--mode, --severity) still override the policy. On pull_request events GitHub runs the workflow from the PR, so a PR could edit those flags too. Branch protection on .github/ (CODEOWNERS) is the usual control there.

How did you verify your code works?

  • New test/altimate/review-verdict-trust.test.ts (11 tests):
    • origin classification, including unknown tools
    • 1 catalog critical + 3 catalog warnings ā†’ COMMENT
    • blockingRules opt-in, and engine criticals and warning patterns still blocking
    • origin covered by the signature (tamper fails verification)
    • policy hash and change listing
    • on real git repos with GITHUB_BASE_REF: a PR relaxing mode/exclude/ai still judged in gate mode with the change shown; a PR adding review.yml not applied; an unchanged policy applied with no note; allowForceTier: false rejecting --force-tier
  • 1,883 tests across the review files and release gates pass. bun typecheck is clean.

Cost side effects

None. One extra local git show per run.

Screenshots / recordings

N/A (CLI).

Checklist

  • I have tested my changes locally
  • I have not included unrelated changes in this PR
  • I checked for cost side effects (model calls, queries or cloud resources that grow with user data) and described them, or wrote "none"

šŸ¤– Generated with Claude Code

Summary by cubic

Makes the review verdict deterministic and tamper-proof: only the deterministic engine can block, and a PR can no longer relax the policy it is reviewed under.

Old behavior: any three non-AI warnings, or one regex-catalog critical, produced REQUEST_CHANGES, and .altimate/review.yml was read from the PR's own checkout. Now pattern-rule criticals and warning counts only comment unless rubric.blockingRules opts a catalog rule in, and the policy comes from the merge-base commit so a PR's edits are listed but only take effect after merge. allowForceTier: false in the base policy rejects --force-tier.

Behavior changes

  • Findings carry an origin (engine | catalog | ai) derived from evidence.tool, stamped in the envelope and covered by the signature; unknown tools count as catalog, so a new detector can't block until classified.
  • Only engine findings count toward REQUEST_CHANGES; the AI lane never blocks.
  • A PR's changes to review.yml show in the comment and policy.changedInPr, apply after merge, and the applied policy's source and hash are signed into the envelope. Non-git callers still read the working tree.
  • CLI flags in the workflow file still override the policy on pull_request events.