Keep `CodeMode(dynamic_catalog=True)` discoveries from rewriting the cached prompt

Merged
DO
9d ago
+617-134

Builds on #10909 (merged), which keeps the ids of the instructions a wrapper toolset passes on, so CodeModeToolset only keys its own catalog block.

With CodeMode(dynamic_catalog=True), finding a deferred tool changed the prompt prefix on every provider:

  • The tool was declared natively (with defer_loading) until found. It was then dropped from tools when it moved into run_code.
  • On OpenAI Responses, dropping it emptied the native tool search's corpus. The optional native tool_search then left the request and the local search_tools took its place, which moved run_code up.
  • The catalog instruction block had no id, so it was rewritten in place instead of appended.

tests/harness/test_history_append_only.py recorded all of this as KNOWN['CodeMode(dynamic_catalog=True)'].

[!WARNING]
Compatibility impact: an agent that has its own toolset with the id code_mode that contributes instructions, and that uses CodeMode(dynamic_catalog=True) (or CodeModeToolset(dynamic_catalog=True)), now fails with a UserError: "Two toolsets have the same id 'code_mode' and both contribute instructions". CodeModeToolset takes the id code_mode only with dynamic_catalog=True, to key its catalog block as toolset:code_mode:catalog. Without dynamic_catalog its id stays None, as before.

Why this can ship in a minor release: this is a deliberate compatibility impact, approved by a maintainer (DouweM), on pydantic-ai-harness, which is still 0.x. It only affects agents that combine the opt-in dynamic_catalog with a toolset of their own named code_mode that has instructions. The failure is an immediate, explicit error on the first request that names the id, not a silent change in behavior. The alternative, leaving the catalog unaddressable, is what made every discovery rewrite the cached prompt.

Migration: rename your own code_mode toolset, for example FunctionToolset(..., id='my_code_tools').

Also: discovery announcements are now recognized only by their SystemPromptPart.metadata. The text-based fallback for histories recorded before announcements carried metadata is removed: a client could trigger it by sending back announcement-shaped system text. In such an older history, a tool found again is announced once more. That's harmless.

Design

  • Sandboxed deferred tools stay off the wire until found. With dynamic_catalog=True, a deferred tool that the tools selector sandboxes is never declared natively. The model finds it through the local search_tools, or by loading the capability that owns it. From then on it can only be called from run_code, so no tool declaration changes. Its native search corpus is then empty, so the optional native tool_search is never sent and the Responses swap goes away.

  • The catalog appends. The catalog block is InstructionPart(name='catalog', on_change='append'). CodeModeToolset.id is 'code_mode' only with dynamic_catalog=True, which keys the block as toolset:code_mode:catalog. With dynamic_catalog=False the id stays None, as before. A discovery appends the updated catalog to history as an InstructionDeltaPart. The block is returned even while it's empty, which renders as nothing, so the first discovery appends too.

  • Trust decision for the catalog (DouweM's). The catalog renders tool definitions into instructions, including the names, descriptions and schemas an MCP server supplies. Tool definitions are trusted operator configuration. The model already receives them in the request's tool definitions with the same authority, so moving them into the catalog, or appending a catalog update, gives them no authority they didn't have. The untrusted-instruction-text rule is about tool results and client or model content, not definitions. The docs say this and advise connecting only MCP servers you trust.

  • Id collision (compatibility impact, see the warning above). With dynamic_catalog=True, another toolset on the agent whose id is code_mode and that contributes instructions now raises the existing duplicate-id UserError. test_catalog_key_collides_with_a_users_code_mode_toolset covers it.

  • When a tool still has to be declared. Two setups still declare such a tool natively, because only the provider's native search can find it there:

    • the 'bm25'/'regex' strategies, which have no local search_tools;
    • a selector that keeps another deferred tool native. That keeps the native search on the wire, and the adapter then drops search_tools.

    These setups behave as before (_place_hidden_tools), and the docs say so.

  • Announcements name functions only by their Python identifier. The discovery announcement used to put the tool name as found into a SystemPromptPart, escaping only </system>. It now lists only the identifier run_code calls ([A-Za-z0-9_], e.g. get_weather for get-weather), so a remote name can't carry backticks, line breaks or prose into system voice. The names as found stay in metadata['tools'], which is how earlier announcements are recognized. Tests cover backtick, newline and instruction-shaped names.

  • Announcements only list what run_code actually gains. CodeModeToolset._functions_revealed_by() works out, from the current step's tools, which found tools the next step sandboxes and under which function name. It applies the same eligibility checks, the same selector and the same first-listed-wins rule for sanitized-name collisions as get_tools. A found tool that the selector keeps native isn't announced. Neither is one whose name sanitizes to a function another tool keeps. Both cases have regression tests.

  • Only Tool Search announces. Loading a capability appends just the catalog update. The docs said every discovery announced, and are corrected.

  • Selector stability. The cache stays intact only if a callable tools selector answers the same for a tool before and after it's revealed. The docs say so. test_a_selector_that_flips_on_reveal_moves_the_tools_prefix pins the move a flipping selector causes.

  • Wrapped toolsets keep their instruction ids. CodeModeToolset.get_instructions builds on super().get_instructions(ctx). With #10909, the parts it relays keep their authors' ids, so only the catalog is keyed by the wrapper. Without #10909, giving the wrapper an id would have handed every anonymous instruction part of the wrapped toolsets the wrapper's key. Harness no longer relies on core's private instruction-collection internals for this.

  • The default mode (dynamic_catalog=False) is unchanged. Its docs already say a discovery rewrites run_code's description.

Why the id is on the toolset, not the capability. A capability key would read better, but the catalog is computed by CodeModeToolset.get_tools for the current step. Capability instructions are resolved with a RunContext built before ToolManager.for_run_step, so a capability-side block would read the previous step's tools. CodeModeToolset is also public and usable without the capability, with its own dynamic_catalog field. Keeping the block on the toolset avoids both problems.

Interaction with #9405 (merged). #9405 records native tools that appear mid-run as ToolAvailabilityDeltaParts. A sandboxed tool that appears mid-run, through a dynamic toolset or MCP list_changed, goes into run_code and appends to the catalog instead, so tools doesn't change and there's nothing for #9405 to record. The append-only checks pass with #9405 merged in.

Expected conflicts: #9512 and #10146 (@mpfaffenberger) both change the native/sandbox split in CodeModeToolset.get_tools.

Tests

  • KNOWN['CodeMode(dynamic_catalog=True)'] is removed. The case is append-only on Anthropic, OpenAI Responses and OpenAI Chat, with no ALLOWED row.
  • New TestDynamicCatalog cases, each a real Agent run:
    • a tool found by search_tools and then called from run_code;
    • the issue's load_capability reproduction. Both check that every request sends identical tools and instructions, that exactly one toolset:code_mode:catalog delta is recorded, and that the run_code result is correct;
    • the two setups that still declare the tool natively;
    • the wrapped toolsets' instruction ids being preserved;
    • the sandbox-name announcement.

Checklist

  • Any AI generated code has been reviewed line-by-line by the human PR author, who stands by it.
  • No breaking changes in accordance with the version policy.
  • Any permitted compatibility impact has a label, warning, migration, release note, and exact API-check waiver.
  • PR title is fit for the release changelog.

🤖 Generated with Claude Code