feat(service): spoken turns on the agent runtime
PR 2 of the Unified agents rollout (STAG-322, "Unifying Voice and Coding Agents"). It builds on #4160, now merged.
The coder runtime (apps/dev) now takes and runs a spoken turn. A call speaks into an agent's own conversation, and the agent's tools and words follow the kind of turn. The brain (apps/luke) is unchanged; PR 7 deletes it.
What changed
The door (coder-host/channel.ts)
CODER_DEPLOYMENT_TURNSadmits a spoken ask for open and follow-up, under the existingx-luke-turn: spokenheader, as the brain's table does.- The coder deployment presents itself (
CODER_DEPLOYMENT_PRESENCE = DEPLOYMENT_PRESENCE.DEPLOYMENT), as the brain's does.- A spoken ask is a principal of the deployment's type. eve never steers it into a typed turn running under the developer's bearer: it waits behind that turn and opens a spoken turn of its own.
- The deployment's
GET …/streamon/eve/coderis admitted, so the voice follow can wake on it. - Cancel and retire are now minted as the deployment acting for the account, as on the brain. eve's cancel and reset routes read nothing of the principal.
- Typed messages still present the account, unchanged.
The turn kind is carried into the turn (coder-host/host.ts)
host.turnKind(auth)readsluke:turnexactly as the brain host does.- The relay records a spoken turn's origin as
spoken. - On a spoken turn, the relay gets the ask record (
askRecord()), as the brain's does. Each ask eve delivered learns the turn it ran in, the transcript's line is tied to that turn (RECEIVED_LINE.TRANSCRIPT), and a Stop the ask took while waiting is carried through/eve/coder. A typed turn takes no ask seam, as before.
Tools follow the turn kind (agent-runtime/tools.ts, coder-host/tool-set.ts, apps/dev/tools/)
queue_question,withdraw_questionandend_callserve the coder role in the registry. They are offered on spoken turns only, throughturnToolsOf(turn)and three new dynamic tool files, one implementation each with the brain (tasks/queue-question.ts,tasks/end-call.ts).sleep,task_wait,delegateandedit_planare unchanged on every turn. See "Deviation from the brief" below.
One prompt with a spoken section (coder-host/turn-context.ts, apps/dev/instructions/prompt.ts)
- The session prompt (
instructions.ts) is untouched. The turn's own instruction (host.turnInstructions) now adds, on a spoken turn, the call's words adapted fromBRAIN_INSTRUCTIONS:- every word is heard, so speak plainly and never in Markdown
- say a sentence before slow work
- summarise, never read out
- speech never waits: no
sleeportask_wait,delegatewithbackground: true - the question queue, facts versus decisions
- when to
end_call
- An agent on a task also gets "Talking the plan through": the plan as a spec, the template's sections and bars, interfaces, hard parts and tests, an unsure developer, what a note holds, and when the plan is done.
Standing context (agent-runtime/standing-context.ts)
- The brain's
brainStandingContextmoved, byte for byte, into a sharedstandingContext. The brain imports it from there. - On every spoken turn, the coder hands the task as it stands: time and zone, the plan document, recent agent edits, sections the developer wrote, open threads, and the board. The coder's own chat digest follows it, so chat is not handed twice.
- On a typed turn, it hands the same only where a revision of the plan by someone other than the agent was saved after the agent's previous turn started (
listRevisions,savedByAnotherSince). A save made while that turn ran is handed too, and so is a teammate's change the agent then wrote over in the same section. It never hands it on an agent's first turn, and never for the agent's own notes.
Deviation from the brief (decided with the orchestrator: option A)
eve 0.74 cannot vary a workflow tool by turn, and its docs say so: "Workflow bodies are for static tools under agent/tools/, not tools returned from defineDynamic resolvers". eve also adds task_wait and task_cancel beside any task tool, with no way to withhold them; the brain is offered task_wait today for the same reason. So, without patching eve:
sleepandtask_waitare still offered on spoken turns. The spoken words tell the model never to call them. Note that eve's own task block still says "If you have nothing to tell the person until a result arrives, call task_wait"; ours is the more specific instruction.delegatealready answers with a receipt on every turn (aservetask). On spoken turns the words saybackground: true, never wait.edit_planstays the synchronous save on both kinds of turn: one transaction of milliseconds. PR 3's singleedit_plansettles its final shape.
Acceptance as built: a spoken turn gets the call's tools and words. sleep and task_wait are told against rather than withheld. A typed turn behaves as today, except that it is handed the task when someone else saved the plan since its previous turn started, as the brief asks.
Measurements
Tokens are counted with o200k_base on the exact text and JSON each runtime builds. The coder agent is on a task, with Chat on and CI readable. Neither column counts connection tools or the standing context, which are the same for both.
| Brain today | Coder, spoken turn | Coder, typed turn | |
|---|---|---|---|
| System prompt (session) | 7,272 | 6,229 | 6,229 |
| Spoken words + talking the plan through | (in the above) | 1,922 | – |
| eve task block | 237 | 237 | 237 |
| Tool definitions | 12,787 (39 tools) | 15,778 (48 tools) | 15,251 (45 tools) |
| Total | 20,296 | 24,166 (+19%) | 21,717 |
A taskless agent's spoken words are 697 tokens. Chat off takes about 630 tokens off the coder's prompt.
Prompt caching is the real latency risk, more than the 3.9k extra tokens. eve merges the session prompt and every turn's system-role instructions into one system message ahead of the history (mergeSystemInstructions). The standing context changes every turn, since it carries the time and the plan. So on each spoken turn the prompt prefix diverges before the history, and the agent's whole history is prefilled uncached. The brain pays this every turn too, but its history is one call's talk. A coding agent mid-PR may carry 50k–200k tokens of history. A typed turn keeps its cache unless it is handed the task, the same as a chat digest today.
The fix I'd recommend (follow-up, medium):
- Keep the spoken words as system role. They are identical on every spoken turn, so consecutive spoken turns keep the cache.
- Hand the standing context as a user-role turn instruction (
defineInstructions({ role: "user" })), which eve appends to history. - Hand it only where it changed since it was last handed, with the time line always. That bounds the growth of the history.
Time to first token was not measured. apps/dev/evals/coder.eval.ts needs a real Postgres (DATABASE_URL) and runs the scripted model, so it would time eve's harness, not a provider's prefill. The resolvers a spoken turn adds at its start are one agent read plus a task, board and comment-thread read, run concurrently. They are small against the prefill above.
Follow-up: "a call joins an agent's running turn"
For the follow-up PR, which starts from main once 2 and 4 merge. Here a spoken ask waits behind a running typed turn. To let it join that turn instead:
- Presence. Present the account for a spoken ask on the coder channel again, so eve steers it into the running turn. Separately, admit the deployment's stream read on
/eve/coderwith a flag of its own; today only a self-presenting actor gets streams. - The shared relay's received line (
brain-host/relay.ts). Today a line is classified by the turn's kind, set at its start. A spoken line joining a typed turn would be written as a typed user row (eve's composed question), duplicating the transcript row and never attaching it. A typed line joining a spoken turn would go down the transcript path, and its awaiting row would never be taken. Classify each line instead:- Take the awaiting row first.
takeAwaitingLineruns under the conversation lock, so it waits out the ask'sdispatchOncecommit. - Then bind the line's joining deliveries again. If an ask binds, it is the transcript's line; otherwise write the user row.
- Without that ordering,
#joined's bind can race the dispatch's delivery write.
- Take the awaiting row first.
- Stop scoped to the spoken part. The relay carries a bound ask's Stop by cancelling its turn (
stoppedOn→stopTurn). A spoken ask that joined an hour-long typed turn and was then stopped on the call would cancel the whole coding turn. - Asks seam on every turn. The coder relay's ask binding then runs on typed turns too, since any typed turn can be joined: two more queries per turn start.
- PR 4's follow projects replies only from the steps after the join.
Tests
pnpm check:affected ran out of memory as one run in this 8 GB sandbox, so its checks ran in parts, each typecheck alone. Lint, knip, discarded-effect, every affected typecheck, and the affected test suites pass. The one exception was the desktop's app-menu.test.ts, which failed once on the pre-rebase merge state; the desktop does not depend on this change. CI is the gate.
coder-host.test.ts:- A spoken turn opens with the call's words, the plan talk and the task as it stands, and is offered the three call tools. A fresh agent's typed turn gets none of it.
- A typed turn is handed the task only after someone else saved the plan after its previous turn started, including where the agent then wrote over the same section. It is never handed it on the first turn or for the agent's own notes.
- A spoken ask dispatched into an agent's session runs as a spoken turn: the ask learns its turn, the transcript's line is the turn's own, and the queued question is journaled. This fails without the ask seam.
coder-agent.test.tsandeve-deployment-secret.test.ts: the coder door admits a spoken ask, opening and following up, and the stream read, as the deployment acting for the account. Cancel and retire are the deployment's own.
Made with Luke