fix(agent): keep the test title out of the system prompt so prompt caching spans tests
Merged
OK
5d ago+107+107+012345678901234567890123456789-81-81-01234567890123456789
#1323 put Test: <title path> into the agent's trusted context, and that context is part of the system prompt. The system prompt now changes from test to test, so the provider prompt cache stops matching across tests. On a live Sonnet 5.5 run, the cost of the same 8 tests went from $0.13 on 0.21.2 to $0.22 on main. OpenAI's promptCacheKey hashes the system prompt, so it now varies per test too. #1323 is unreleased, so this fixes it before #1210 publishes.
- The title moves into the step's user prompt. The built-in agent's act prompt (
packages/e2e/src/agent/default-agent.ts) and every judgment prompt (buildPromptinpackages/e2e/src/agent/prompts.ts) now open withTest: billing > upgrades the tier. The system prompt is the same for every test, as it was in 0.21.2. ctx.agentContextgoes back to the agent'scontextplus the test'sagentContext. Custom executors read the title from the newctx.attempt.titlePath, which is redacted per step like the context.- The
keyedContextsplit from #1323 is gone. The cache key digestsagentContextagain, which is the same value it digested before, so committed recordings keep replaying. - The explore prompts now point at the
Test:line for the goal. #1323 put the goal there, and the prompts still said it was in "project context". - The #1323 changeset is reworded to describe the final behavior.
Not covered: @e2e-dev/decision's executor reads ctx.agentContext and never had the title. The changeset scopes the line to the built-in agent.
Verified
Ran it locally: yes
- web benchmark,
tests-agent/control-inventory.e2e.ts+login-form.e2e.ts(8 tests),anthropic/claude-sonnet-5.5with adaptive thinking,--no-cache --workers 1, the same config and port on both builds:
| main | this branch | |
|---|---|---|
| tests | 8/8 | 8/8 |
| tokens | 197.9k | 197.9k |
| cached | 61% | 84% |
| cost | $0.22 | $0.11 |
main: AI 197.9k tokens · 61% cached · $0.22 · 21 model calls · gateway/anthropic/claude-sonnet-5.5
branch: AI 197.9k tokens · 84% cached · $0.11 · 21 model calls · gateway/anthropic/claude-sonnet-5.5
- Strict replay of the committed web-benchmark recordings on this branch (
--strict-cache, the default model) gaveCache 26 replayed · 2 handed off · 4 missedand 33/33 passed. That matches main, with noREPLAY_STALE. - New unit tests show that the act and judgment prompts open with the title, that two tests with different titles send an identical system prompt, and that a secret value in a title is redacted in the act prompt (
packages/e2e/tests/unit/agent-entry-prompt.test.ts).