feat(voice,service,host): type into a live call

Merged
DA
5d ago
+904-28

PR 6 of the Unified agents rollout (STAG-322), stacked on #4300 (5b, feat/calls-are-on-agents).

What the prompt asked

Dean decided that a line typed while a call is up on an agent is answered aloud. Before this PR, a typed line always steered the agent (steerCodingAgent), so its reply never reached the voice. This PR:

  • Hands a line typed during that agent's call to the call. The host decides this, never the voice window.
  • Records the line once in the transcript, as typed.
  • Leaves a line typed with no call, or with a call on another agent, as an ordinary steer.

A deviation from the brief: the voice doesn't choose between answering a typed line and delegating it

The brief asked the host to append the line to the Live session "as the developer's own text item, delegation_id: null", so that the voice would treat it as speech. GPT-Live has no way to take a user text item under client delegation. A running session takes only session.instructions|thinking|commentary.append. For typed input, OpenAI's delegation guide says to "send the typed value directly to the backend" and mirror it into the session. There's a second problem: LiveSessionService builds every ask from the spoken-transcript ledger. Even if the voice delegated on an appended line, its delegation would be held back until the next spoken words arrived.

So the agent always answers a typed line, and the voice speaks the reply. The orchestrator approved this; it reads "answered aloud" this way. Every append stays on the trusted side (packages/live/AGENTS.md).

How it works

  • Host (compose-coding-agents.ts, compose-host.ts):
    • CODING_AGENTS_MESSAGE first offers the line to LiveSessionHolder.typeLine(agentId, text). The holder now keeps the call's agentId, which the host checked against the open task when the call was created.
    • When a started call on that agent takes the line, the host answers the tab with the agent as it stands and doesn't steer.
    • Otherwise the line steers exactly as before: no call, a call on another agent, a call not yet started, or a line of files alone.
  • Wire (live-contract.ts): a new device frame session.typed {text}, trimmed and bounded like a message to an agent. The service reads it as a report and forwards it nowhere, beside session.stop (frames.ts, service.ts).
  • Exchange (LiveSessionService.typeLine):
    1. Flushes pending spoken rows, then writes the line once as the developer's typed row. The row's client id is minted for its ask, and it names the call (voice_session_id) and the instant on the call's clock it was typed at (from_ms). It is written behind every earlier transcript write.
    2. Hands the line to the agent through the existing spoken submitAsk, at the session's next revision. It joins a running turn, and Stop and the follow work as for speech. It supersedes an older exchange, and the next delegation supersedes it. The relay writes no second row: the turn picks up the typed row through attachAskLines by the ask's client id.
    3. Registers the exchange with delegation_id: null, so the agent's reply is spoken session-wide.
    4. Tells the voice in a session.thinking.append (null) that the developer typed the line and that Luke will answer it, so the voice neither answers nor delegates it itself.
    5. Claims no ledger row, so no later spoken delegation carries the line again.

The three fixes from the investigation

  • Recovery after re-attach returns no delegation. asks.spokenIn now reads typed from the typed row under the ask's client id, and recoverRuns returns delegationId: null for those runs. LiveBrainRecoveredRun.delegationId is now a LiveDelegationId.
  • The question is never the bare line. The agent is handed the transcript since the last ask, then the line under its own lead. On top of that, a call's typed line (one that names a voice_session_id) is never taken as a line awaiting a turn (findAwaitingLines), and doesn't read the agent as running (newestAwaitingLines).
  • The spoken reply is linked to the typed ask. latestSpokenLine now answers a typed call line's own client id as its ask, so Luke's words after it carry that delegation_id and fold into the turn as for speech. spokenRowsOn (a call's transcript lines) reads channel = voice only, so the typed line is never drawn as a spoken line.

TYPED_ASK_METADATA gains optional voice_session_id and from_ms. Only a line typed into a call carries them.

From review

  • A Retry is one line. The composer's send key travels on session.typed, and the line is written and asked under sentLineId(clientKey), the same id a steer of that send would write. A Retry after a lost answer is the same line on either path.
  • The spoken context is claimed by the typed ask. The developer's words since the last ask, which the typed ask carries as context, become its rows, as a delegation's do. Neither the hang-up nor the next delegation hands them on again.
  • The answer reads running. The host answers a line handed to the call with the agent running, as a steer's answer does.
  • Known limitation, left for a follow-up. A typed frame that reaches the service just as the voice function recycles can be lost, as session.stop can be today: the voice socket has no acknowledgment or replay of the device's frames.

Mentions and attachments

  • Mentions: the voice's thinking note writes them as their labels (linkLabels), since the voice can't use an address. The typed row and the agent's ask keep the line as typed, with its [@label](url) links:
    • The tab reads a sent line back by its words, so the row has to match the composer's words or the line would hang as pending.
    • The agent can follow a link, as it does for any typed line.
  • Attachments are dropped on a line typed into a call. The call hands on words alone: nothing on the call path can carry a file to the agent, and the voice can't read one. A line of files alone, with no words, steers as before, with its files.

Docs

  • packages/service/README.md (the device frames) and PRIVACY.md (what reaches OpenAI on a call) describe the typed line.

Tests

Each of these fails without the change.

  • live-session-service.test.ts:
    • A typed line is written once as typed under its ask's id.
    • Its question is more than the bare line.
    • The voice gets a thinking note with mentions as labels, and nothing is said of the line itself.
    • The reply is spoken with delegation_id: null.
    • It takes the next revision, silencing the older exchange, and the next delegation silences it and never carries it.
    • Nothing is taken before the session starts.
    • A recovered typed run speaks session-wide.
  • live-session-holder.test.ts: only a started call on the same agent takes the line. Another agent's line, a line before start, and a line after the call ends are not taken. The window never appends.
  • compose-coding-agents.test.ts:
    • A line typed during that agent's call goes to the call and isn't steered.
    • No call, a call on another agent, or files alone steer as before.
  • live-contract.test.ts and frames.test.ts: the session.typed frame is a bounded, trimmed report.
  • voice-writer.test.ts: one typed row naming the call, idempotent under its id, and the voice's reply after it carries the typed ask's id.
  • hosted-asks.test.ts: spokenIn reads which ask a typed line made.
  • coding-agent-store.test.ts: a typed call line is never taken by its words as an awaiting line.

Ran locally: the voice, service voice/tasks, host, features, wire, and database model suites, plus hosted-store-writer and hosted-asks (189 files pass). biome and oxlint are clean on the change.

Made with Luke


Devin Review