Send a question's chosen preview back to Claude Code (annotations)

Merged
TE
11d ago
+541-19

Leo's ask, his words: "ask user question tool and its various states need looking at allegedly there's some sort of vibe-coded thing that agents can do to have prettier ask question situations...? idk but we could look into that/research that then support it..."

Leo-ask: ask-question-states

What was wrong

Claude Code's AskUserQuestion lets an agent attach a preview (a mockup or code sample) to each option. The answer the host sends back may carry annotations (question text -> {preview?, notes?}), and the agent then reads "Q"="A" selected preview: <mockup>, so it knows which mockup the person picked. cmux sent only answers. The card never sent the chosen preview, the pane relay refused any annotations param, and acpmux would have dropped it anyway.

The fix

  • Card (question/model.ts reply): for a Claude Code ask, each single-select item whose chosen option has a preview adds annotations[prompt] = {preview}. Other answers, multi-selects, the Chief and Codex send nothing new. App.tsx and direct.ts pass it through to _acpmux/permission_respond.
  • Pane relay, in both Swift AcpmuxPaneMethods and cmux-agent-pane-policy, from the same policy.json: annotations is a known param. It goes only beside answers, to a pending question the daemon sent, keyed by that question's own items, and each value is {preview?, notes?} strings of at most 64 KiB.
  • acpmux (questions::check_annotations, respond_permission_annotated):
    • It takes annotations only from the person's own connection, and only with answers. A phone's signed answer or a session's answer can't carry them.
    • For Claude Code's AskUserQuestion, a preview must match one of that question's option previews as normalized, and the agent gets its own original text back, since Claude Code refuses a preview it didn't write. Notes must be 1 to 8192 bytes. Any other field, or any other question, is refused.
    • For any other question, the annotations are dropped, so the answer still goes through.
  • Protocol IR, generated TS types and acpmux-schema.json document the param.

Not in this PR: notes UI (the protocol carries notes now; the card doesn't offer a field yet). HTML previews, markdown descriptions, answered and skipped summaries, and MCP elicitation belong to the transcript-parity lane.

Relay analysis

This adds one param to an existing method of the agent pane relay (`_acpmux/permission_respond`). It doesn't add a method, and it changes nothing remote: `RemoteAcpmuxWire` still passes only the option.

  • Command or content execution: none. The value is checked text that ends up in Claude Code's own tool input, and Claude Code validates it again (`admitCardAnswer`).
  • Unowned objects: the relay accepts it only for a question the daemon sent this pane, keyed by that question's items. acpmux accepts it only from the person's connection, for the pending prompt, beside answers.
  • Local-state exposure: none. Nothing is read back. A preview has to be text the agent itself wrote, and notes are capped at 8192 bytes.

Test

Red at the test-only commit 3fc192c7d1c130e92cf4eb2578cc9d28026e33c9, green at the fix:

  • bun test src/agent-session/acpmux/question/model.test.ts src/agent-session/acpmux/question/record.test.ts (from webviews/, on cmux16): red 11 pass / 1 fail ("a Claude answer sends the chosen option's preview"), green 12 pass / 0 fail.
  • acpmux questions_annotations* (tests/hub_integration/questions.rs: carried with the agent's own preview text; refused when the preview isn't the agent's, the key or field is unknown, notes are empty or too long, or answers are missing; dropped for a question that isn't Claude's) and the policy cases in tests/cases/check.json run on the fleet with cmux-tui-rust-check.sh test. Results are below.
  • cmux-ci compile-check --app at the head: see below.

Observed / Not run

Observed: the webview tests above on cmux16. Also at the head on cmux16: question/, needsYou/ and test/question-card-submit.test.tsx all green, tsc --noEmit clean, and prepush-gate.py reports no new violations.

Not run: a live Claude Code run that reads the annotation back (the fake agent echoes the tool input instead), and the Swift relay unit tests (the shared policy cases cover the same rule in Rust; the Swift harness doesn't read questions state yet).

No capture: no visible UI change; the card looks the same and sends one more field.

Gallery: agent-pane.question-card

Changelog

Changed: When you answer an agent's question that has option previews, Claude Code now learns which preview you picked.

Adopted and rebased (2026-10-25)

The stale-pr-reconciler, word for word: "These PRs in your area have no live owner (Leo, 2026-10-25: "these should never have been this stale for this long"). Adopt each: pr-owner claim N (in that repo), rebase onto its base, an independent review at the new head, then land it".

  • Squashed onto feat-cmux-next (it was 3,151 commits behind). Previews are now {text, format} objects on base; the card already reads preview.text, and check_annotations compares against the trimmed raw option text, so the agent still gets its own text back. The only conflict was where the new function sits in hub/questions.rs.
  • Codex review (gpt-6-astra xhigh) round 1, BLOCK: 1. a selected preview over 65,536 UTF-8 bytes rode along as an annotation, and both pane relays then refused the whole answer. Fixed: the card leaves out a preview over the relay's bound (MAX_ANNOTATION_BYTES, the same as policy.json maximum_annotation_bytes), and the answer still goes through. Test first: a preview over the relay's 65,536-byte bound is left out, and the answer still goes (32,768 x "é" is exactly the bound; one byte more, and 65,537 ASCII bytes, are left out).
  • Round 2: SHIP at f7c5b2337d7. Round 3 (a vp fmt reformat of that test only): SHIP at 079ae3ee5a2.
  • Rebased again onto 714af0ca098, past #25268 (chat multiplayer: author, answeredBy). respond_permission_annotated now returns (pending, answered_by, annotations), and the request handler keeps base's answerer.author.
  • Round 4 (fresh reviewer), BLOCK: 1. the annotation checks ran before the pending lookup, so a losing annotated answer from a non-person connection got person_required instead of -32002 with the first answer's answeredBy/outcome. Fixed by moving them to after the lookup. Test first: a_losing_answer_is_told_who_answered_first now also sends an annotated losing answer (red at 2eddd4787f0, green at 2551dc935a6, hub_integration questions/permission/author 40 pass).
  • Round 5: SHIP at 2551dc935a6. The Release compile (Xcode 26) passed at that head. The Swift test reds are in untouched areas (password manager route, icon family, sidebar rail), and the generated-files red is base's open-remote CLI name (#25092).

Tests on the rebased branch (fast-checks on Blacksmith)

  • Red at the test commit 2838a9b7da8: questions_annotations_carry... and ..._refused_and_the_ask_stays fail; policy full_check_order fails on the new cases.
  • Green at fb4cc7c16b3: acpmux --test hub_integration questions 8 pass, --lib questions 5 pass, cmux-agent-pane-policy parity 16 pass; clippy -D warnings and fmt clean. cmux-agent-pane-policy --test typescript the_allowlist_is_what_the_page_sends fails the same way on base de67de98d17 (_acpmux/mcp/reconnect), so it isn't from this PR.
  • Webviews: red at a8986604eb9 (the bound test fails), green at f7c5b2337d7: question/, needsYou/, test/question-card-submit.test.tsx 46 pass, tsc --noEmit clean.
  • Base reds this PR shows too: Script tests (release-compile-derived-data.test.sh "bench OTHER_SWIFT_FLAGS", fails on base 5e6cb133c60); gallery health agent-pane.composer#file-not-attached (broken on other PRs too).

No timers or observers added. No capture: the card looks the same and sends one more field.

Agent session: big-red
🤖 Generated with Claude Code