feat: implement issue #1787 — calibrate solution-architect escalation so conforming changes are not escalated
Summary
Adds escalation calibration to the solution-architect advisory prompt, using the worked-example shape prompts/qa-lead/advisory.md already uses. It also makes the risk tier an explicit line in the comment body. New bats checks guard the calibration against a later edit or template sync.
It also fixes the ADR path in step 2. The prompt said cat docs/architecture/adr/ADR-NNNN.md, but the files are named NNNN-<slug>.md (for example 0002-channel-tag-release-and-rings.md), so the example never resolved. Step 2 now shows the real naming and how a file maps to its ADR-NNNN citation.
Closes #1787
Problem
#1787: solution-architect scored 0.625 against the 0.7 gate. All three held-out failures were wrong escalate decisions on otherwise correct advisories: the persona cited the right ADR, then escalated a change that fits it.
The prompt had only one generic line ("do not manufacture concern") and nothing tying a class of structural finding to a tier and an escalate decision. Two of the failures also lost points because the risk tier was never stated.
Calibration added to step 4 of the prompt:
| Situation | Risk tier | Escalate | Alignment |
|---|---|---|---|
| Conforms to an accepted ADR | LOW | no | aligned; "Nothing structural" is a complete answer |
| Drifts, and the ADR already names the conforming alternative | MEDIUM | no | drifts; the recommendation is the remedy |
| No recorded ADR governs | MEDIUM if it sets a boundary, LOW if trivial | no | name the gap, recommend an ADR |
| Merging breaks a decided contract beyond this PR | HIGH | yes | — |
| Deliberately overturns an accepted ADR | by blast radius, usually MEDIUM | yes | only a maintainer can supersede an ADR |
Two consistency rules are also added: HIGH always pairs with escalate = yes, and an advisory rated "aligned" must not escalate.
The examples are written at class level. They don't restate any held-out case, so the prompt adapts to the classes the cases exercise rather than to the cases themselves (AC7).
Risk
Low. This changes the prompt text of an advisory-only persona that can't block or write anything. The runner path is unchanged (AC5): the sentinels, the <!-- persona:solution-architect --> marker, the read-only gh fetches and the "cite an ADR or say there is none" rule are all as before. No eval case, judge or threshold is touched (AC7).
Test plan
bats tests/test_advisory_offline_mode.bats: 18/18 pass. The 6 new checks cover:- the calibration section exists inside Steps;
- a conforming change is LOW / no;
- "no ADR governs" is not an escalation;
- HIGH pairs with escalate = yes;
- the body shape includes
**Risk tier:**; - step 2's example ADR path names a file that exists in
docs/architecture/adr/.
- Mutation check: run against the earlier prompt, each new check fails.
- The 5 calibration checks fail with
main's prompt. - The ADR-path check fails with the
ADR-NNNN.mdpath. - The existing checks pass either way.
- The 5 calibration checks fail with
bats tests/test_skill_evals.bats(45),tests/test_validate_cases.bats(21) andtests/test_score_gate.bats(30) all pass.- No shell scripts changed.
prompts/is excluded from markdownlint by config. - Not run here: the live re-baseline that AC6 asks for. See Monitoring.
Rollback
Revert this PR. There are no non-revertible side effects.
Monitoring
- Run
skill-eval-report.ymlforsolution-architectand record the score beside0.25 → 0.625 → 0.875 → ?(AC6).- 0.875 is the 2026-10-04 run in #2063, taken without any prompt change.
- Both #1787 regression cases must pass:
sa-hold-eval-immutability-gate-no-adrandsa-hold-repin-channel-to-frozen-tag. - The remaining #2063 failure,
sa-hold-onboard-advisory-persona-router, wasgot null(no parsed output) rather than a wrong escalation. Watch whether it recovers.
- Eval-health issue #2063 auto-closes on recovery.
Interaction contract
- N/A — this PR does not add or change an agentic role. It changes the prompt text of an existing persona, with no change to its trigger, contract or runtime.
Checklist
-
shellcheck scripts/*.shpasses for any changed shell scripts (none changed). - Docs updated where behavior or standards changed (the prompt is the persona's behaviour spec).
Related: #2203. That issue would make persona escalations hold PRs, so this calibration should land first.
🤖 Generated with Claude Code