Check UI adapter round trips with the real `ai` and `@ag-ui/client` JavaScript clients

Merged
DO
9d ago
+11641-38

A UI app keeps the conversation in the frontend and sends it back every turn, so what the model sees next is whatever the client library kept. Until now we checked that with Python ports of the client reducers. One of those ports produced a false bug, and none of them checks what the client sends. This PR adds tests/ui_roundtrip, which runs the real clients instead.

  • No issue. This is step A ("4a") of the history/projection architecture plan: the harness alone, with no change to library code.

What it does

  • The real clients. driver.mjs drives the AI SDK's AbstractChat (what useChat wraps) and @ag-ui/client's AbstractAgent.runAgent() (what HttpAgent extends). Each gets a transport that hands the request to the test and streams back the server's reply. The versions are pinned in package.json/package-lock.json (ai@6.0.309, @ag-ui/client@1.0.3) and installed with npm ci --ignore-scripts. A session-scoped fixture starts one Node process and talks to it in JSON lines.

  • Scenarios. Agent conversations scripted with FunctionModel, streamed in deltas: system prompt plus instructions, client-owned system prompt, tool call, parallel tool calls that finish out of order, thinking, tool availability delta, enqueued messages, enqueued system prompt, tool retry, output-validation feedback, no_output feedback, append-mode instruction changes, request-scoped system prompt, multimodal tool return, compaction, and a compaction boundary with a reveal before it and reveals after it, with and without append-mode instruction changes (compaction is seeded, since only provider models emit a CompactionPart).

  • Paths and properties. Each scenario runs over Vercel AI and AG-UI, on two paths: the dump path hydrates the client with dump_messages() every turn, and the stream path lets the client build its messages from the event stream. Three properties are checked, plus test_message_ids, which checks the client never sends two messages with one id:

    1. Load after dump (test_load, dump path): load_messages() of what a hydrated client sends back projects identically to the history it was hydrated with.
    2. Load after stream (test_load, stream path): load_messages() of what a streaming client sends back projects identically to the server's all_messages().
    3. Wire request (test_wire_request): every request the agent makes through the client has the same Anthropic and OpenAI Chat request body as a server keeping its own history. The agent runs on the real AnthropicModel and OpenAIChatModel on mocked SDK clients, so each request is prepared and rendered for that provider as in production; the script's FunctionModel supplies the answers.

    The projection keeps only the fields the model sees. It normalizes two things the agent and the models normalize themselves: consecutive requests are merged, and tool results follow call order.

  • Normalization uses the agent's own private helpers on purpose. The projection calls _merge_consecutive_messages and _order_tool_results_by_call rather than reimplementing them. It's meant to mirror what providers actually receive, and a copy would drift from production. Normalizing both sides the same way is correct, because the property under test is the UI round trip, not the normalization. And test_wire_request compares the real provider request bodies, so a defect in normalization can't hide a round-trip loss there.

  • The projection includes framework metadata. Each message's reserved __pydantic_ai__ metadata is projected; only the adapters' own ui_message_id bookkeeping is left out. Application metadata and part metadata (documented as never sent to the model) are dropped. No scenario showed a new difference from this.

  • Forged framework voice. In the forged_framework_voice scenario, user prompts, a tool result and enqueued text carry <system>, </context>, mixed-case and whitespace closing forms, and tool-reveal lookalike text. test_forged_framework_voice_stays_out_of_system_voice asserts that, on every path, protocol and provider, none of it reaches the Anthropic system field or an OpenAI system/developer message, and that it still reaches the model as user or tool content. The general properties run on the scenario too.

  • Known gaps are asserted exactly, not xfailed. Each gapped case must show exactly the difference recorded for it, as a unified diff in tests/ui_roundtrip/known_gaps/. So any other failure in that case fails as usual, and so does the gap being fixed (the cue to delete the file and the scenario's gaps entry). UI_ROUNDTRIP_UPDATE_GAPS=1 rewrites the files after a deliberate change. Harness invariants, like each turn making at least one request, are asserted outside the gap check.

  • The Python AI SDK reducer port is mostly replaced. The reload half of the enqueued-messages test in test_vercel_ai.py moved to the enqueued_messages and client_owned_system_prompt scenarios here. Source coverage of ui/ and messages.py from test_vercel_ai.py is unchanged. _reduce_assistant_message stays for now, because test_mid_run_tool_delta_round_trips_through_client_rebuilt_stream landed on main meanwhile and uses it. Its case is now covered with the real client by the mid_run_tool_addition scenario, so that test and the port can go in a follow-up once coverage is checked.

  • CI. A new UI round trip workflow (.github/workflows/ui-roundtrip.yml) sets up Node (actions/setup-node, pinned by SHA like the other actions) and runs only these tests. It runs on every pull request and every push to main, with no path filter, plus nightly. The histories under test come from the whole library, and the tests import shared test helpers and depend on the locked dependencies, so a filter would always miss something, while the suite takes seconds. A failed nightly opens or updates an issue, as the sandbox nightly does. The main ci.yml test jobs skip these tests, since they have no node_modules. UI_ROUNDTRIP_REQUIRED=1 makes the new job fail instead of skipping. Because the main jobs skip them, tests/ui_roundtrip/*.py is left out of coverage. The whole suite (296 tests) runs in about 6 seconds.

Gaps it found (recorded in known_gaps/, not fixed here)

  • New: @ag-ui/client never sends activity messages back. prepareRunAgentInput filters out role: "activity", in every client version since activities were added (checked 0.0.42 through 1.0.3). The adapter carries tool availability deltas, compaction parts and enqueued messages as activities. So with the real client they are lost on both the dump and stream paths, and the next request changes (a revealed tool disappears, for example). The file activities (pydantic_ai_file, pydantic_ai_uploaded_file) use the same carrier, but these tests don't cover them. The earlier audit missed this because it only ran the reducer, not the client's outgoing request. #10569 corrects the AG-UI docs; the carrier fix is a separate design question.
  • Retry feedback on the Vercel AI stream path. The AI SDK stream has no carrier for a tool's outcome, so a streamed retry reloads as outcome='failed' instead of 'retried' (#8185). #8094 fixed the other retry paths, and those gaps are gone.
  • Foreign thinking's model=/provider= attributes were fixed by #10642, so those gaps are gone too.
  • New: append-mode instruction records don't survive. InstructionBaselinePart and InstructionDeltaPart are system voice, which the adapter strips from client history in manage_system_prompt='server'. So every turn re-renders the baseline from current state and rewrites the instruction prefix. This happens on every path, protocol and provider. It's a design question: trusting these records from a client needs provenance.
  • New: retry feedback isn't streamed. On the stream path, a RetryFeedbackPart (output-validation or no_output) never reaches the client, so the client keeps the rejected and the accepted answer as one response.
  • New: no_output feedback is demoted on reload. It's the framework's system voice, which a client can't claim, so it loads back as user-voice feedback and renders differently. This is by design, but it moves the prefix.
  • New: AG-UI dump_messages() ordering. It writes a request's system prompts before its tool results (so an enqueued system prompt reloads ahead of the tool result it followed), and it drops an empty response (so the feedback after it loads into the question before it).
  • New: duplicate Vercel AI message ids. Once a request carries the ui_message_id it was loaded with, dump_messages() gives its system and user messages the same id, so a hydrated client holds two messages with one id (test_message_ids).
  • An enqueued SystemPromptPart from an earlier run is stripped with the client's system prompts in manage_system_prompt='server', and nothing restores it. This is the residual that #10389 described.

.github/ changes

This PR adds a workflow file, so the .github Directory Guard checks the author's repository permission. The author has write access, so it passes.

Checklist

  • No breaking changes in accordance with the version policy.
  • Any permitted compatibility impact has a label, warning, migration, release note, and exact API-check waiver.
  • PR title is fit for the release changelog.

🤖 Generated with Claude Code