Anchor the tool-callability evidence window to the response's serving provider

Merged
DO
3mo ago
+510-43

Follows #7271, which makes availability gate execution — that's what turns this window's conservatism into a wrong refusal rather than a harmless one. Also carries the refusal-message rewording DouweM asked for on #7271, which landed just after it merged.

The problem

post_compaction_window is provider-agnostic by construction: at parse time there is no "current" provider, so a CompactionPart some other provider would skip on the wire still counts as a boundary. That was fine while under-counting only cost a redundant, idempotent re-disclosure. Once availability gates execution, under-counting also refuses a call — and it does so wrongly in exactly the cases where the wire never honored the boundary:

  1. a foreign-provider part (the trim is same-provider only, so the serving adapter replays the full history) — FallbackModel switching providers mid-conversation is the common way this happens;
  2. a same-provider OpenAI part without encrypted_content — both the trim and the render skip it;
  3. a content-less Anthropic part from a failed compaction — a documented server-side no-op.

In all three the model can still see the schema, so it is refused a tool that is genuinely in its context, and pays a turn regenerating the search or load.

The fix

Anchor the dispatch-time evidence window to the serving response's own provenance. The check runs once a response exists, and ModelResponse.provider_name records which provider served it — a fact, not a prediction, so FallbackModel's "which sub-model will handle the next request" ambiguity never arises. _post_compaction_window_for_response counts a boundary only if that provider would have honored it on the wire, and only if it sits strictly before the serving response (a provider can emit a compaction part and keep generating in the same response, but that response was not part of the request whose tools the model saw). No provider name on the response falls back to the provider-agnostic window.

The window's consumers split by which side of the request boundary their output feeds:

  • Retrospective — consumed immediately, never re-read. The callability gate is the only member; an exact per-response window is safe precisely because the verdict doesn't outlive the dispatch, and over-refusal is the harmful direction.
  • Prospective — lands in a future request: re-advertisement, instructions, search ranking, catalogs. These keep the conservative provider-agnostic window, since a future model that does honor the boundary must still be shown the omitted schemas. Over-disclosure is redundant but benign. Tool search needs no change.

is_tool_available stays the single predicate; only the evidence it reads is anchored, via a private AnchoredEvidence on RunContext set at dispatch. Introspection between requests still errs conservative.

That evidence is additive rather than a corrected copy of discovered_tool_names/loaded_capability_ids, and it has to be: those sets are shared mutable run state that build_run_context passes by reference and that tool execution writes in-step reveals into, so handing the dispatch context its own widened copies would silently drop those writes.

Shared wire-boundary predicate

_compaction_part_is_wire_boundary now answers "would this provider honor this part on the wire" for both the gate and _trim_messages_before_compaction, subsuming the trim's requires_encrypted_content knob so the two layers can't drift.

That knob stays a parameter rather than a lookup on provider_name: one adapter serves many provider names — an Azure-backed OpenAIResponsesModel reports system='azure' — so a name table would treat an unrenderable part as a trim boundary and drop the pre-boundary history with nothing sent in its place. test_openai_responses_non_openai_provider_name_still_requires_encrypted_content pins that.

Unifying the two also fixes a latent trim bug: previously a content-less, payload-less Anthropic CompactionPart (a failed compaction) trimmed the history behind it even though the wire treats it as a no-op. It no longer does — test_anthropic_contentless_compaction_does_not_trim.

The gate's default reading (any payload counts) and an adapter's render condition can still disagree for a part stamped with an encrypted-mode provider that carries only plaintext; there the gate errs toward refusing rather than toward admitting an unseen tool. Unifying those properly wants the declared per-model compaction mode from #7255, which is being done separately.

Interaction with #7271's load-is-the-reveal shortcut

Where the two features meet there's a case neither covers alone: a capability loaded before a boundary the wire skipped, whose reveal delta history processing has also dropped. Both halves of the agnostic evidence are then gone, and the shortcut is the only thing that can answer — but only if it reads the anchored window. test_stripped_reveal_marker_survives_a_boundary_the_wire_skipped pins it; without either half of that union the tool is refused with no way for the model to regenerate the evidence.

Durability

AnchoredEvidence is dispatch-time run state a tool can read through is_tool_available, so it crosses both boundaries that carry such state: serialized in TemporalRunContext (both exhaustiveness guards enforce this, and a payload that predates the field defaults to empty rather than raising) and projected into the Prefect cache key alongside loaded_capability_ids/discovered_tool_names, so two runs differing only in it can't replay each other's tool results.

One thing worth a close look

In CallToolsNode, for_run_step deliberately returns the same manager when the run step is unchanged, so the evidence computed for dispatch is assigned onto its existing context rather than arriving through a fresh one. Replacing tool_manager.ctx wholesale would discard the retries state that same-step return exists to preserve, so only the one private field is assigned. It's the narrow option and still the part I'd most want read closely, though a reviewer has since walked both for_run_step outcomes and reached the same conclusion.

Checklist

  • Any AI generated code has been reviewed line-by-line by the human PR author, who stands by it.
  • No breaking changes in accordance with the version policy.
  • PR title is fit for the release changelog.