ci(cmux-next): path-based risk tiers with a required evidence check

Closed
TE
19d ago
+670-0

Pull requests into feat-cmux-next now get a risk:low, risk:medium or risk:high label from the paths they change, and a risk evidence check that stays red until the description has the evidence that tier needs. The policy is in docs/ci/risk-tiers.md, and CLAUDE.md tells every agent to follow it.

TierPathsEvidence before merge
lowdocs, tests, skills, plans, CI and toolinggreen CI
mediumapp UI, SDK bindings, anything unclassifiedplus a screenshot or clip under ## Proof, or Not UI: <why>
highdaemon, acpmux, protocol and spec, persistence, migrations, schemas/, typing path, remote, cloud and browser import, signing, release and publish, vendored pins, and the merge gates themselvesplus Review: naming the reviewed commit and Runtime test: naming the run that executed the change

Why now: 2,049 pull requests merged into feat-cmux-next from 2026-10-04 to 2026-10-18. 932 had a failing check on their merged head, not counting review bots and report-only lanes. cmux-next swift test runs on a pull request only with full-ci, never on a push to the branch (.github/workflows/cmux-next.yml:1760), and in the last 25 runs that executed it, 24 failed. So a new Swift test usually compiles and never runs before merge. On that history the classifier grades 380 pull requests low, 992 medium and 677 high.

How it works:

  • scripts/ci/risk_tier.py holds the ordered path rules. A pull request takes the highest tier of any file. An unknown path is medium, never low. A medium change over 1,000 lines outside tests and generated files becomes high.
  • .github/workflows/risk-tier.yml runs on pull_request_target, so fork pull requests get labels too. It checks out only the base's classifier and reads the file list, commit list and description through the API. It never runs the pull request's code. It re-runs when the description is edited, so adding the evidence turns the check green.
  • Review: must name a commit on the pull request. Runtime test: must name a run URL, or a commit on the pull request that a tagged build or Testbox ran. Proof must be media, not a link to a CI run.
  • A maintainer can add risk:waived. It counts only when the timeline shows an admin or maintainer added it. Agents are told never to.
  • The unit tests run in their own workflow, and the grading job has no if:, so no skipped risk evidence run can become the newest check and read as a pass. A newer event cancels an older grading run, and gh-merge-green refuses a cancelled check.
  • manaflow-ai/cmux#21692 (on main, where the installed helper lives) makes gh-merge-green refuse a red risk evidence check even with --override, the same way it treats god-file and l10n lint.

The policy also states two practices that already have tooling: ship user-visible high-tier behavior behind a setting that defaults off, and treat nightly-next as the canary, reverting with the existing gh-merge-green --revert when a high-tier change breaks it and the fix misses the next cut.

Because this workflow runs from the base, it starts labeling only after it merges. Open pull requests pick up a label on their next push or description edit.

Testing

  • python3 tests/test_risk_tier.py: 17 tests pass. They cover path rules, the size bump, the evidence parser, template comments and bare links not counting as proof, values on the next line, stale review and runtime SHAs, and the waiver labeler.
  • python3 scripts/ci/risk_tier.py pr --repo manaflow-ai/cmux --number N against live #21334 (medium), #21645 (medium), #21646 (low) and #21690 (medium) printed the expected tiers and missing evidence. That run read the API and wrote nothing.
  • bash tests/test_ci_self_hosted_guard.sh accepts both jobs' runners.
  • python3 scripts/verify-local.py --affected origin/feat-cmux-next: everything passes except localization parity, which fails the same way (34,264 errors) at the tip without this change.
  • bash scripts/ci/guards-local.sh: 100 steps pass and 19 fail, the same 19 as at the tip 4b1b0235f55 without this change.
  • Not run: the workflow itself. pull_request_target workflows run from the base, so the first real label comes after merge. actionlint isn't installed on this host.

This pull request grades itself high, because it changes a merge gate.

Review: BLOCK at af6ab7a3f26 (a skipped run could hide a red check, plus classifier and parser gaps), then SHIP with nits at 9132ed0ed5e. That was a cmux-review subagent, not the author's agent. 2397770200 only adds the waiver-lookup nit from that review.
Runtime test: python3 tests/test_risk_tier.py and risk_tier.py pr against live #21334, #21645, #21646 and #21690, run at 2397770200.

Follow-ups from review, not in this pull request: grading any workflow with pull_request_target or a write token as high needs the file's contents, not only its path; and Runtime test: accepts any run URL without tying it to this pull request.

Changelog

none

Proof

Not UI: adds a CI label, a check and docs. Nothing in the app changes.

🤖 Generated with Claude Code