feat(pstack): shorten /ketchup replies and build them on real visuals
Why
/ketchup replies run long, and they seldom use a real visual. In a blind eval, the current skill gives a median of 736 text words on Claude Opus 5.5 (1,530 on long chats). Only 8 of 15 replies have a real visual. Most of its "visuals" are Markdown tables, which the user reads as text.
What changed
pstack/skills/ketchup/SKILL.md- The reply text stays under 250 words. Markdown tables count as text. A visual replaces the text it shows.
- One status visual shows each PR, run, or agent the chat touched, with its state. Under it are at most three one-line results. A result keeps its key number unless the visual shows it.
- Each action item has three lines: the decision, the recommendation and why, and what happens with no answer. A before-and-after visual holds the options and their costs.
- A new "Visuals" section defines a visual as a picture: a screenshot, a video, a Mermaid diagram, a
/visualizechart, or a generated image. A Markdown table is not a visual. - Before the reply, the agent checks each number it will show against the facts it comes from, and says so when it corrects one.
- A visible change that has no screenshot goes into the status visual, marked "No screenshot yet", so the three-result limit cannot drop it.
pstack/README.mdandpstack/skills/poteto-help/SKILL.md: the/ketchuprow matches the new description.
Scope
- The scope rules do not change: the turn stays read-only, the action-item definition stays, and names stay the ones the user knows.
- No other skill changes.
/brostays as it is.
Blast Radius
/ketchup sets disable-model-invocation: true, so it loads only when a user types it. The skill grows from 490 to 606 words, about 150 more input tokens per call. The median Opus reply drops from 741 to 365 words with diagram labels counted. A surface that does not render Mermaid shows the diagram source.
Verification
Preregistered blind eval on 5 invented chats (2 long, 3 short). The same deterministic grader scores every arm. The skill, grader, gates, and seed were frozen by hash before the first run of each round. Round 2 tested this PR without the last two bullets of "What changed". Round 3 tested the text in this PR: 15 Opus 5.5 high runs, the same model and effort as round 2, against the same 15 before runs.
| Opus gate | Before | Round 3 (this PR) | Round 2 (without the fix) | Result |
|---|---|---|---|---|
| Median text words ≤ 250 | 736 | 213 (95% CI 199 to 221) | 195 | pass |
| Long chats: after / before ≤ 0.50 | 1,529.5 words | 234.5 words, ratio 0.153 | 0.156 | pass |
| Real visual in ≥ 80% of runs | 8/15 | 15/15 | 15/15 | pass |
| Every after run names every action item | 15/15 | 15/15 | 15/15 | pass |
| Must-cover recall ≥ before − 0.05 | 0.956 | 0.950 (diff CI −0.039 to +0.022) | 0.972 | pass |
Two-catch check. A separate eval found two catches that the old skill makes: it corrects contrast ratios that disagree with the colors, and it says when a visible change has no screenshot. On those two chats, with 3 Opus 5.5 xhigh runs per arm, this PR without the fix dropped both. The fix gets both back.
| Opus 5.5 xhigh | Contrast corrected | "No screenshot" said |
|---|---|---|
Old skill (main) | 3/3 | 3/3 |
| This PR without the fix | 0/3 | 2/3 |
| This PR | 3/3 | 3/3 |
At high effort, the old skill corrects the contrast in 0 of 6 runs and this PR in 5 of 6. In round 3, the "No screenshot yet" mark is in 9 of 9 runs where a visible change has no screenshot (old skill 7/9).
- Hand audit of every graded miss: by hand, all 15 round 3 runs cover every must-cover fact. The four grader misses are wording misses. No round 3 run invents a number or links a file that does not exist.
- Known gaps: 13 of 15 round 3 runs repeat one or two numbers in both the text and a visual. One run of 15 has 302 text words; gate 1 is the median. The number check sometimes corrects a small number too ("about 3.5 screens" becomes 3.44).
- Round 2 also ran Grok 4.7 high on the text without the fix, reported only: 730 to 153 median text words, 2/5 to 5/5 real visuals. Round 3 did not rerun Grok.
quick-validate.mjs,lint-skill-frontmatter.mjs, andvalidate-plugins.mjspass.
Round 2 renders (the text without the fix):
Long chat: before and after replies rendered
Short chat: before and after replies rendered
