Keep deferred tools searchable after discovery and reset reveal state at `CompactionPart` boundaries

Merged
DO
3mo ago
+584-29

Follow-up to #7104 (based on that branch; retargets to main when it merges). After compaction, the model "starts anew": this PR makes derived tool state agree with what the model can actually see, in two independent, mutually reinforcing changes.

1. The search corpus never shrinks with discovery

ToolSearchToolset used to subtract already-revealed names from the corpus, so a tool discovered before a CompactionPart became unsearchable exactly when the model could no longer remember discovering it — search_tools (and any custom search_fn, as in the report) answered "No matching tools found" for a tool that exists. Now every searchable deferred tool stays in the corpus for the whole run; the default keyword strategy ranks not-yet-revealed matches first when trimming to max_results (as suggested by @pingxia23 on #7204). Rediscovery is idempotent — parse_discovered_tools is a set union — and a fresh search return regenerates reveal evidence the model can see, so discovery is self-healing under any history truncation: server-side compaction, history processors, or harness compaction strategies.

2. A CompactionPart is a visibility boundary for derived state

parse_discovered_tools and parse_loaded_capabilities now reset their accumulated state at every CompactionPart, at its exact part position (Anthropic ignores earlier parts within the same response too). Without this, on #7104's lazy-advertisement wire designs the runtime kept claiming "revealed" for tools whose schema carrier (additional_tools item / replay block) sits before the boundary — schema-withheld in tools, carrier invisible to the model, and unsearchable per the old corpus rule. The reset is deliberately provider-agnostic: over-counting blocks rediscovery and ships invisible "reveals", while under-counting merely permits a redundant, idempotent re-search or capability re-load. Tools remain callable throughout — the reset hides schemas, it never revokes execution.

3. The boundary is public API: pydantic_ai.messages.post_compaction_window

Capabilities keep needing the same rule — harness CodeMode's dynamic-catalog announcements and DynamicWorkflow's reveal announcements both dedupe against in-memory state that compaction silently invalidates (pydantic-ai-harness#562, pydantic-ai-harness#563). post_compaction_window(messages) returns the messages from the latest CompactionPart onward at part-level precision; both parsers here are rebuilt on it as the single definition of the boundary (the reverse boundary-locate keeps the scan cheap), and the compaction docs tell capability authors to derive model-visible state from the window rather than instance attributes. The default search ranking also puts undiscovered matches unconditionally first now, so an already-available tool never displaces an undiscovered match under max_results — and test_repeated_searches_paginate_through_a_large_corpus pins that repeating a query still enumerates a large corpus page by page (the explicit pagination signal is #7232).

4. A capability's own tools always activate its validate/execute hooks

Flagged by Veria on this PR and confirmed by reproduction: a capability-owned tool stays callable while its capability counts as unloaded, and CombinedCapability's loaded-gate then skipped the owner's before/after/wrap_tool_validate/_execute hooks — approval gates included — for that call. The boundary reset made this reachable in legitimate runs (load pair pre-boundary, model calls the tool from summary memory); the fabricated-history route (reveal evidence with no load pair) already had the same bypass on the base branch, and its regression test fails there too. The tool-scoped dispatchers now force the owning capability's hooks for its own tool (_ctx_for_tool_hook), with capability_loaded kept truthfully False; cross-cutting hooks of unrelated unloaded capabilities remain skipped as before.

Verified against the live APIs

  • Anthropic (docs + live probe): "when the API receives a compaction block, all content blocks before it are ignored" — confirmed semantically (planted marker unrecoverable) and in billing (identical input_tokens with and without pre-boundary content). Server-side tool_search re-discovers a deferred tool after compaction from the request's declarations, and the API accepts a dangling pre-boundary reveal without error — the boundary-reset semantics here match the server's own.
  • OpenAI (live probe): input items before a compaction item are not ignored — a planted marker that cannot be in the encrypted state was recovered, and the items are billed. For this PR that only makes the reset err on the benign side; the resulting inefficiency in OpenAICompaction's stateful full-replay path is filed separately as #7224. (Stateless mode already replaces history client-side and is unaffected.)

Tests

Each fix is independently load-bearing (mutation-verified: reinstating the corpus subtraction or neutering either reset fails tests):

  • the reporter's exact partial-corpus scenario: A–C discovered, compaction, custom search_fn still sees A–E and A is rediscovered and executed
  • corpus invariance with no compaction at all, and undiscovered-first ordering when max_results trims
  • part-level boundary reset for all four discovery representations (typed, native, legacy metadata, delta) and for capability load pairs, including a pair split across the boundary
  • capability tools re-hide after the boundary and re-reveal on a fresh load_capability
  • a pre-boundary-revealed tool called directly post-compaction (no re-search) still executes
  • a capability-owned tool called post-compaction (and via fabricated reveal with no load at all) runs its owner's execute hooks with capability_loaded=False
  • on the OpenAI wire, a pre-boundary reveal resolves as withheld again and emits no additional_tools carrier

Checklist

  • Any AI generated code has been reviewed line-by-line by the human PR author, who stands by it.
  • No breaking changes in accordance with the version policy.
  • PR title is fit for the release changelog.