Chat: send images and PDFs to acpmux ahead of the prompt, by reference

Closed
TE
10d ago
+582-34

Summary

  • Images and PDFs now go to acpmux ahead of their prompt, and the prompt names them by reference, so no prompt frame carries image bytes.
  • New acpmux method _acpmux/attachment_put {sessionId, mimeType, data} keeps an image or PDF (at most 10 MiB; a larger one still goes inline) by SHA-256 under <ACPMUX_HOME>/attachments, the same store prompt images already use. It marks the file as that chat's and answers {id}.
  • A prompt block names a kept file as uri: "acpmux-attachment:<id>" with no data. acpmux puts the bytes back before anything else reads the prompt, so user_message image ids, _acpmux/attachment reads, PDF handling and harness delivery are unchanged. A prompt that names an id its own chat did not put is refused with attachment.missing; it is not run without the image.
  • Host (Swift AcpmuxPaneMethods, plus policy.json): the put is allowed only for a chat in the pane's scope, with exactly sessionId, mimeType and data, and needs no gesture. It runs nothing; the prompt still needs its gesture. The relay already strips _meta inside prompt blocks, which is why the reference uses ACP's uri field.
  • Page (direct.ts putAttachments, attachments.ts promptBlocks): it puts only when the daemon lists _acpmux/attachment_put in initialize. Images go inline as before for an older acpmux, a peer's acpmux without the method, or a put that fails.

Why

On 2026-10-20 a coordinator chat lost its connection. A prompt with six Retina screenshots was about 15 MB of base64, and the next one passed acpmux's 16 MiB WebSocket frame limit. acpmux reset the socket, and the chat said "Not sent: The agent connection was interrupted". The product call is never to refuse a person's screenshots and to send attachments out of band, so prompt size never hits a frame limit. Related PRs:

  • #23027: raises the frame limit to 64 MiB, logs a reset, and auto-reconnects and auto-resends (the backstop).
  • #23041: fits every image to 2000 px and 1.5 MiB.
  • This PR: removes the images from the prompt frame. Prompt frames are now small, so the host's prompt journal (8 MiB cap) can keep image prompts for resend too.

Tests (committed red before the fix)

  • cmux-tui/crates/acpmux/tests/prompt_images.rs an_attachment_put_ahead_goes_with_the_prompt_that_names_it:
    • a put answers the image's SHA-256, and a non-image type is refused;
    • a prompt naming the id runs, its user_message names the image, and _acpmux/attachment reads its bytes back;
    • another chat's prompt naming the same id is refused with attachment.missing, and nothing is logged for it.
  • AgentPaneAttachmentRelayTests.anImageIsPutOnlyForTheChatInTheShapeThePaneSends: refuses a chat outside the pane's scope and an extra param, and passes the pane's own shape without a gesture.
  • webviews/src/agent-session/acpmux/attachmentPut.test.ts:
    • promptBlocks names a kept image by reference;
    • on the mock acpmux, a prompt with two screenshots sends two puts and a prompt frame without the bytes;
    • an acpmux without the method gets the image inline;
    • a prompt sent after one still putting its images is written after it;
    • a prompt removed from the queue while its images are put is never sent (the pane withdraws it, since acpmux has no such prompt yet).
  • The mock acpmux now lists the method and expands references, so the Playwright sent-images test exercises the whole path.
  • Not run locally: Rust, Swift and bun tests are not run on this Mac. CI runs them.

Notes

  • The method is added to the Swift allowlists and the Rust mirror policy.json alike; their lists match.
  • Nothing garbage-collects attachments/ today (prompt images on base have none either); the put marks are empty files beside it. A follow-up gives puts the age sweep and purge-on-kill that hub/file_uploads.rs has.

Agent session: cosmic-gray

šŸ¤– Generated with Claude Code


View with [code]smith Autofix with [code]smith
Need help on this PR? Tag @codesmith-bot with what you need.


Summary by cubic

Images and PDFs are sent to acpmux ahead of the prompt that names them, by URI, keeping attachment bytes out of prompt frames and avoiding WebSocket frame-limit disconnects. Prompts reach acpmux in send order, even when earlier attachment puts take longer.

  • _acpmux/attachment_put keeps images and PDFs up to 10 MiB by SHA-256 and marks them for that chat, so the base64 read-back fits one acpmux message.
  • acpmux restores referenced bytes before running the prompt and rejects attachments that chat did not put with attachment.missing.
  • Prompts wait only for their own puts and for a prior prompt whose request is not yet written; a prompt with nothing to put still writes its request in the send call, so a dropped socket fails it like any in-flight request.
  • If the method is unavailable, a file is over the cap, or a put fails or times out after 30 seconds, attachments go inline as before.
  • Withdrawing a prompt still putting its images removes it in the pane and the send drops it; a retry of the same id stays withdrawn, and acpmux drops a copy it already took.