feat(chat-workflow): Anthropic prompt cache control (cutover Bundle A.6)

Merged
SW
5mo ago
+348-1

Summary

Fourth open-agents → api cutover bundle. The workflow's runAgentStep was sending the same system prompt + tool definitions on every turn as fresh input. Anthropic prompt caching can shave ~90% off the input cost of subsequent turns, but only if cache-control markers are attached to messages + tools — which api wasn't doing.

This PR ports open-agents' addCacheControl helper and wires it into runAgentStep via a prepareStep callback (messages) + static tool-wrapping (tools).

Production reproducer (Haiku 4.5, same 2-turn prompt to both)

TURN 1 ('reply OK only')TURN 2 ('now reply DONE only')
api prod (broken)inputTokens=5932 · cacheWrite=0 cacheRead=0 · cost=$0.005952inputTokens=5934 · cacheWrite=0 cacheRead=0 · cost=$0.005959
open-agents prodinputTokens=10969 · cacheWrite=10966 · cost=$0.069inputTokens=10981 · cacheRead=10966 cacheWrite=12 · cost=$0.006 (12× drop)

api's cost is flat across turns (every turn pays full input rate). open-agents drops 12× on turn 2 because the entire system prompt + tools prefix is read from cache.

Changes (SRP)

  • lib/agent/contextManagement/isAnthropicModel.ts — predicate (byte-identical port of open-agents/packages/agent/context-management/cache-control.ts:5)
  • lib/agent/contextManagement/addCacheControlToTools.ts — marks the LAST tool with cacheControl: { type: "ephemeral" } (last-only conserves Anthropic's 4-breakpoint limit)
  • lib/agent/contextManagement/addCacheControlToMessages.ts — marks the LAST message with cacheControl on every step, per Anthropic's "mark the final block of the final message" guidance
  • runAgentStep:
    • Wraps tool set via addCacheControlToTools(...) before passing to streamText (static — set once per step)
    • Adds a prepareStep callback that wraps messages via addCacheControlToMessages(...) on every internal model call

Cutover roadmap

#BundleThis PRNotes
1C per-message cost/usage✅ merged#592 / #593
2B subagent live progress + transcript✅ merged#594 / #595
3A.7 real cwd in system prompt✅ merged#597 / #598
4A.6 Anthropic prompt cache control✅ this PR
5A.4 forward Privy JWT as recoupAccessToken

Test plan

  • 19 new unit tests (7 isAnthropicModel + 5 addCacheControlToTools + 5 addCacheControlToMessages + 2 runAgentStep wiring)
  • Full suite 3138/3138 pass; lint clean
  • E2E on preview: re-run the 2-turn reproducer, confirm cacheWriteTokens > 0 on turn 1 and cacheReadTokens > 0 on turn 2 (matching open-agents' shape)

🤖 Generated with Claude Code


Summary by cubic

Adds Anthropic prompt caching to the chat workflow by tagging the final message and last tool with cacheControl: { type: "ephemeral" }. This lets Claude reuse the system prompt and tools across turns, cutting input tokens and cost on later turns (~10–12× in our repro).

  • New Features
    • Added isAnthropicModel, addCacheControlToMessages, and addCacheControlToTools helpers (ports from open-agents).
    • Updated runAgentStep to wrap tools and use prepareStep to tag only the last message/tool for Anthropic models (respects cache breakpoint limits).
    • No-op for non-Anthropic models; 19 unit tests added.