feat(voice,service,host): type into a live call
PR 6 of the Unified agents rollout (STAG-322), stacked on #4300 (5b, feat/calls-are-on-agents).
What the prompt asked
Dean decided that a line typed while a call is up on an agent is answered aloud. Before this PR, a typed line always steered the agent (steerCodingAgent), so its reply never reached the voice. This PR:
- Hands a line typed during that agent's call to the call. The host decides this, never the voice window.
- Records the line once in the transcript, as typed.
- Leaves a line typed with no call, or with a call on another agent, as an ordinary steer.
A deviation from the brief: the voice doesn't choose between answering a typed line and delegating it
The brief asked the host to append the line to the Live session "as the developer's own text item, delegation_id: null", so that the voice would treat it as speech. GPT-Live has no way to take a user text item under client delegation. A running session takes only session.instructions|thinking|commentary.append. For typed input, OpenAI's delegation guide says to "send the typed value directly to the backend" and mirror it into the session. There's a second problem: LiveSessionService builds every ask from the spoken-transcript ledger. Even if the voice delegated on an appended line, its delegation would be held back until the next spoken words arrived.
So the agent always answers a typed line, and the voice speaks the reply. The orchestrator approved this; it reads "answered aloud" this way. Every append stays on the trusted side (packages/live/AGENTS.md).
How it works
- Host (
compose-coding-agents.ts,compose-host.ts):CODING_AGENTS_MESSAGEfirst offers the line toLiveSessionHolder.typeLine(agentId, text). The holder now keeps the call'sagentId, which the host checked against the open task when the call was created.- When a started call on that agent takes the line, the host answers the tab with the agent as it stands and doesn't steer.
- Otherwise the line steers exactly as before: no call, a call on another agent, a call not yet started, or a line of files alone.
- Wire (
live-contract.ts): a new device framesession.typed {text}, trimmed and bounded like a message to an agent. The service reads it as a report and forwards it nowhere, besidesession.stop(frames.ts,service.ts). - Exchange (
LiveSessionService.typeLine):- Flushes pending spoken rows, then writes the line once as the developer's typed row. The row's client id is minted for its ask, and it names the call (
voice_session_id) and the instant on the call's clock it was typed at (from_ms). It is written behind every earlier transcript write. - Hands the line to the agent through the existing spoken
submitAsk, at the session's next revision. It joins a running turn, and Stop and the follow work as for speech. It supersedes an older exchange, and the next delegation supersedes it. The relay writes no second row: the turn picks up the typed row throughattachAskLinesby the ask's client id. - Registers the exchange with
delegation_id: null, so the agent's reply is spoken session-wide. - Tells the voice in a
session.thinking.append(null) that the developer typed the line and that Luke will answer it, so the voice neither answers nor delegates it itself. - Claims no ledger row, so no later spoken delegation carries the line again.
- Flushes pending spoken rows, then writes the line once as the developer's typed row. The row's client id is minted for its ask, and it names the call (
The three fixes from the investigation
- Recovery after re-attach returns no delegation.
asks.spokenInnow readstypedfrom the typed row under the ask's client id, andrecoverRunsreturnsdelegationId: nullfor those runs.LiveBrainRecoveredRun.delegationIdis now aLiveDelegationId. - The question is never the bare line. The agent is handed the transcript since the last ask, then the line under its own lead. On top of that, a call's typed line (one that names a
voice_session_id) is never taken as a line awaiting a turn (findAwaitingLines), and doesn't read the agent as running (newestAwaitingLines). - The spoken reply is linked to the typed ask.
latestSpokenLinenow answers a typed call line's own client id as its ask, so Luke's words after it carry thatdelegation_idand fold into the turn as for speech.spokenRowsOn(a call's transcript lines) readschannel = voiceonly, so the typed line is never drawn as a spoken line.
TYPED_ASK_METADATA gains optional voice_session_id and from_ms. Only a line typed into a call carries them.
From review
- A Retry is one line. The composer's send key travels on
session.typed, and the line is written and asked undersentLineId(clientKey), the same id a steer of that send would write. A Retry after a lost answer is the same line on either path. - The spoken context is claimed by the typed ask. The developer's words since the last ask, which the typed ask carries as context, become its rows, as a delegation's do. Neither the hang-up nor the next delegation hands them on again.
- The answer reads running. The host answers a line handed to the call with the agent running, as a steer's answer does.
- Known limitation, left for a follow-up. A typed frame that reaches the service just as the voice function recycles can be lost, as
session.stopcan be today: the voice socket has no acknowledgment or replay of the device's frames.
Mentions and attachments
- Mentions: the voice's thinking note writes them as their labels (
linkLabels), since the voice can't use an address. The typed row and the agent's ask keep the line as typed, with its[@label](url)links:- The tab reads a sent line back by its words, so the row has to match the composer's words or the line would hang as pending.
- The agent can follow a link, as it does for any typed line.
- Attachments are dropped on a line typed into a call. The call hands on words alone: nothing on the call path can carry a file to the agent, and the voice can't read one. A line of files alone, with no words, steers as before, with its files.
Docs
packages/service/README.md(the device frames) andPRIVACY.md(what reaches OpenAI on a call) describe the typed line.
Tests
Each of these fails without the change.
live-session-service.test.ts:- A typed line is written once as typed under its ask's id.
- Its question is more than the bare line.
- The voice gets a thinking note with mentions as labels, and nothing is said of the line itself.
- The reply is spoken with
delegation_id: null. - It takes the next revision, silencing the older exchange, and the next delegation silences it and never carries it.
- Nothing is taken before the session starts.
- A recovered typed run speaks session-wide.
live-session-holder.test.ts: only a started call on the same agent takes the line. Another agent's line, a line before start, and a line after the call ends are not taken. The window never appends.compose-coding-agents.test.ts:- A line typed during that agent's call goes to the call and isn't steered.
- No call, a call on another agent, or files alone steer as before.
live-contract.test.tsandframes.test.ts: thesession.typedframe is a bounded, trimmed report.voice-writer.test.ts: one typed row naming the call, idempotent under its id, and the voice's reply after it carries the typed ask's id.hosted-asks.test.ts:spokenInreads which ask a typed line made.coding-agent-store.test.ts: a typed call line is never taken by its words as an awaiting line.
Ran locally: the voice, service voice/tasks, host, features, wire, and database model suites, plus hosted-store-writer and hosted-asks (189 files pass). biome and oxlint are clean on the change.
Made with Luke