add anthropic caching to system prompt in agent

Merged
TK
10mo ago
+24-4

Add Anthropic Prompt Caching for Agent System Prompts

Summary

Added Anthropic's ephemeral cache control to system prompts in v3AgentHandler.ts. Refactored into a single prependSystemMessage() function used by both execute() and stream().

Why

Reduces token costs and latency for Claude models by caching the system prompt server-side. The providerOptions are ignored by non-Anthropic models, so this is safe for all providers.

Limitations

Currently limited to the system prompt only. Extending to conversation messages requires restructuring message processing to insert cache breakpoints at stable boundaries.

Test Plan

  • Verify agent works with Claude and non Anthropic models
  • Confirm cached_input_tokens appears in usage metrics with Claude

Summary by cubic

Added Anthropic ephemeral caching to the agent’s system prompt to lower Claude token usage and reduce latency. Safe for non-Anthropic models; the options are ignored.

  • New Features

    • System messages now include Anthropic providerOptions with cacheControl: ephemeral.
  • Refactors

    • Introduced prependSystemMessage() and used it in both text generation and streaming to centralize system prompt handling.