perf(service): an agent's standing context rides as a turn message, so its history stays cached

Merged
DA
5d ago
+502-302

PR 3b of the Unified agents rollout (STAG-322, "Unifying Voice and Coding Agents"), on top of #4183.

Why

#4183 handed a coding agent's per-turn standing context (time and zone, task, plan document, recent agent edits, open threads, board), its chat digest, and the call's words as a system-role turn instruction. It also offered the call's tools only on spoken turns. eve merges every system instruction into the one system message ahead of the history, and the tools lead the request. So every spoken turn, and every switch from typed to spoken, changed the prefix before the history. The agent's whole history (50k–200k tokens mid-PR) was then prefilled uncached.

What changed

  • The system prompt and the tools are the same on every turn, whatever its kind.
    • apps/dev/instructions/prompt.ts now resolves only at session.started, and CoderHost.turnInstructions is removed.
    • queue_question, withdraw_question and end_call are offered on every turn as static tools. Off a call they answer with a plain refusal ("No call is live …", tool-set.ts's runCallTool). end_call ends the turn only when it is accepted.
    • The per-turn-kind gating #4183 added (turnToolsOf, offeredTurnTools, SpokenTurnTool) is deleted. context-estimate.ts counts one tool set.
  • Everything that changes from turn to turn is a user-role message on the turn, after the history. The new apps/dev/instructions/turn-message.ts returns defineInstructions({ role: "user" }) at turn.started. This is eve's own mechanism: a dynamic user result is appended to durable history before the current delivery. No eve patch. It hands CoderHost.turnMessage(admitted, { kind, turnId }, handed):
    • The call's words (#4183's "This turn is spoken" and "Talking the plan through") open the first of a run of spoken turns. A later spoken turn gets one line saying they still hold, and a typed turn after a call gets one line saying they no longer do.
    • The developer's time, on every turn (one line).
    • The task as it stands, only when it changed. That means another writer's plan revision was saved since the agent's previous turn started (#4183's savedByAnotherSince; the agent's own notes never count), or what the model reads of the rest of the task changed since the session last handed it. That rest is the name, brief, linked issues, open threads (openThreadsText) and board (boardText), hashed by taskKey. A typed first turn is not handed it, because its first message already is the plan. The session then hands the whole task once, on its next turn.
    • The chat digest, only when it differs from the one last handed.
  • What a session has handed is kept in eve's durable session state (luke.coder.turn-handed). A compaction (hooks/store.ts, handedAfterCompaction) clears the task, the chat, and whether the call's words stand, so the next turn hands them again instead of relying on a summary that may have dropped them. It keeps the last turn's kind, so a typed turn after a compacted call is still told the words no longer hold.

Provider prompt caching

  • Anthropic: eve already sets cache breakpoints (harness/prompt-cache.js) on any model whose provider contains anthropic. It marks the system message, the last tool, the last assistant message (the end of the history), and the last message. The coder's wrapLanguageModel layers keep provider as anthropic.messages, so the markers apply. All 4 showed up in the measured requests. model.ts needs no change.
  • OpenAI: prefix caching is automatic on the Responses API once the prefix is stable, which this PR provides.

Measurements

No live model or provider key could run here, so these are eve's real requests. The coder app ran under eve eval with the scripted fixture model, against a local Postgres. A throwaway wrapper (not committed) dumped each call's prompt, under provider id anthropic.messages so eve applied its Anthropic markers. The agent's history was about 170k tokens; then one typed turn and two spoken turns ran through the deployment door. Tokens are o200k_base. Cacheable means the prefix a turn's first request shares with the request before it, in Anthropic's order: tools, system, messages.

Turn's first requestBefore (#4183): not cacheable / totalAfter: not cacheable / total
First spoken turn after a typed one172,615 / 185,914 (shares the tools before the call's own)2,486 / 185,371 (the call's words, the task, the last answer)
Second of two consecutive spoken turns169,245 / 185,954 (shares the tools only)153 / 185,551

Tests

coder-host.test.ts. Each test below fails on the code before this PR:

  • The call's words open the first of a run of spoken turns as the turn's message. A later spoken turn gets the "still holds" line, a typed turn after them gets the "no longer holds" line, and the next spoken turn gets the words again.
  • The call's own tools run on a spoken turn and are refused on a typed one.
  • A spoken turn's message hands the task where the session has not handed it, then only the time while nothing changed, then the task again once the task was renamed.
  • A typed turn's message hands the task once after the first turn, then only after another writer's revision. It never hands it for the agent's own notes, and never on the first turn.
  • A chat digest is handed once and not again while it is unchanged.

The system prompt is the session's alone and no turn resolver adds to it, so it is the same on consecutive turns by construction.

That the message reaches the model after the history is eve's user-role contract, and vitest cannot run eve. I checked it end to end under eve eval: system, history, the turn's message, eve's skill announcement, then the new words. apps/dev/evals/coder.eval.ts passes apart from its chat-handle assertion, which expects @agent- handles where chat now writes mention links. That assertion fails the same way without this change.

Locally: the service and coder typechecks, knip, oxlint, and the coder-host, agent-runtime and brain-host suites (338 tests) pass. CI is the gate.

Notes for review

  • turn-message.ts is not a runtime edge. It calls a promise helper in apps/dev/host.ts, as the tool files do, so neither AGENTS.md nor effect-edges.json changes.
  • Each turn now runs one instruction resolver (the message) where #4183 ran one (the system instruction).

Made with Luke