PR #12 - ReviewStage/luke

Summary

Add a portable realtime voice contract in packages/sidecar-core/ that builds the minted session, validates an untrusted client-secret response, turns server events into captions, and selects which attention reviews are worth voicing.
Mint short-lived Realtime credentials in the Electron main process from the standing OPENAI_API_KEY. Only the ephemeral secret crosses the preload bridge, so a compromised renderer cannot outlive the credential it was handed.
Open the WebRTC call from the sandboxed renderer, with push-to-talk conversation, live captions for both speakers, a typed fallback through the same conversation, and proactive speech driven by the attention layer.

Push-to-talk, not an open microphone

The session is minted with turn_detection: null, so nothing is ever decided by a voice-activity heuristic. The microphone track is created once and kept disabled; holding the talk button or the space bar enables it, releasing commits the turn, and Escape abandons it. An abandoned turn sends input_audio_buffer.clear rather than committing, so audio the developer changed their mind about is not inherited by the next turn. A window that loses focus mid-press releases the microphone, which is otherwise the one path that strands it open.

Connecting is deliberately not the same as listening: connect() leaves the track disabled and the panel reports Voice ready rather than Listening.

Any failure — a refused call, a denied microphone, a stalled handshake, a dropped connection — releases the microphone before reporting the failure. failed offers "Start voice" again, so leaving the device held would both light the macOS microphone indicator with no stop path and let a retry stack a second call on a live track.

Proactive speech

AttentionReview already answered two different questions, and this wires up the second one. decision says the session needs attention, which the panel shows; outcome says whether to voice it now. Only decided reviews are spoken — a deduplicated review still marks the session as needing attention, but repeating the same sentence aloud would be noise.

An approved sentence is spoken verbatim rather than regenerated, so what Luke says out loud is exactly the bounded, redacted summary that passed review. Luke never opens the microphone on its own, so an approved sentence that arrives when it cannot be voiced — voice off, or a turn already open — is shown as a caption instead of being silently dropped.

One turn at a time

The API answers one turn at a time, and that is the whole of the arbitration: while a turn is open, nothing starts another. speak, sendText, and startListening report the refusal and the caller shows the sentence instead. The panel disables the talk button and the text field while Luke is answering, so the controls offer exactly what the session will accept.

This replaced a queue-and-defer state machine that produced 10 of this PR's 28 review findings, including the eleventh round's. Details in the simplification note below.

What Luke can see in conversation

The standing instructions describe session state as something Luke knows, so the roster is pushed into the conversation when the call opens and whenever it changes — the same bounded, redacted fields the attention layer already sends (provider, title, status, provider summary). It is added as context with no response.create, so it never makes Luke start talking, and it carries no transcript, file path, or command output. Without it the prompt would assert a capability the connection never provided.

Why is voice off?

"Voice unavailable" had several causes that were indistinguishable from the panel: no API key in the launching process, a fixture run, a rejected mint, an unreachable endpoint, a malformed response, or a clock skewed far enough that a fresh credential arrived expired. The app knew which one it hit and said none of them.

Each mint now records an outcome and a detail. Startup writes one line to stderr, and the expanded panel carries a Voice diagnostics block in development builds — or in a packaged build launched with --diagnostics — reporting whether the key reached the process, whether this is a fixture run, the model, voice, endpoint, microphone permission, and the last mint outcome.

The report carries no credential material by construction: whether a key was found, never the key, and never any part of a minted secret. A test asserts the serialized report cannot contain the key.

Trust boundary

The layer is optional. Without OPENAI_API_KEY, the panel states that voice is unavailable and Luke keeps observing sessions; no product behavior requires credentials.
Fixture and evidence runs never mint a credential, so screenshots stay reproducible offline and without a network.
The standing key never enters the renderer. Logs carry a status code or an error name only — never the key, the minted secret, or session material.
Every failure path — no credential, a refused call, a denied microphone, a malformed mint response, an already-expired secret, a frame outside the protocol — resolves to an explicit unavailable/failed state rather than a guess.
The renderer stays sandboxed with context isolation; the new IPC is one invoke for a credential and one main-to-renderer push for approved speech, both gated on trustedSender.

Variable	Default	Purpose
`OPENAI_API_KEY`	unset	Enables voice when present
`LUKE_REALTIME_MODEL`	`gpt-realtime-2.1`	Model used for the conversation
`LUKE_REALTIME_VOICE`	`marin`	Spoken voice

OPENAI_BASE_URL redirects attention review, which runs entirely in the main process. Voice ignores it and always uses the canonical host, because the conversation must also satisfy the renderer CSP, which permits only https://api.openai.com. Honoring it would mint against one host and fail the SDP exchange against another — an endpoint that appears to work until the first word is spoken.

Panel layout

The expanded panel is a fixed 620x520 box, and the voice panel plus an open diagnostics block overflowed it — pushing the footer, and with it the control that stops a call, off screen with nothing able to scroll.

That turned out to be a pre-existing defect rather than one this branch introduced: on main, five observed sessions was already enough to put "Start microphone" out of reach. It is fixed separately in #13, now merged, which makes the panel body the one scroll container. This branch is rebased on it and adds one commit so the voice panel sits inside that region rather than after it.

Simplification

Ten of the twenty-eight findings on this branch came from one place: a turn-arbitration state machine that is not part of a standard Realtime integration. It queued approved sentences behind a cap, deferred typed messages behind held presses, cancelled in-flight responses and waited for the cancellation to land, and tracked all of it in a flag parallel to the status — which the eleventh round found another way to desynchronise.

56bad11 removes it. Gone: the pending-speech queue with its cap and eviction handback, the pending-request slot and its text handback, response.cancel, the onUnspoken and onUnsentText callbacks, and the #responseActive flag. The status is the turn state, so nothing can drift from it. Two identical input-buffer builders became one.

Net 401 lines removed (489 deletions, 88 insertions) across the session, its tests, and the core contract. Session fields: 12 → 9. Callbacks: 7 → 5.

The cost is barge-in. Talking over Luke is no longer possible. That machinery was built entirely against assumed response.cancel semantics that cannot be checked without a live call, and Luke answers in one or two sentences by design. It can be added deliberately once the real behaviour has been observed — which is the right order.

Evidence

Platform-independent checks: ./scripts/check.sh passed — lint, 3 typecheck projects, 193 tests (5 harness, 53 core, 135 desktop), all builds. Rebased onto main through #19.
Panel layout re-measured after the rebase at 620x520 in headless Chrome: with the voice panel and diagnostics open, the body scrolls (336px visible of 583px content at three sessions, 933px at eight) and the footer stays fully on screen at both
macOS Electron verification (./scripts/verify.sh): portable checks passed, then packaging stopped with error: this command requires macOS in this Linux workspace; CI's macOS app/evidence job covers it
End-to-end run of the real mint path (openAiRealtimeCredentialsFromEnvironment over real fetch) against a local stub of the client-secrets endpoint:

credential handed to the renderer:
{
  "value": "ek_stub_ephemeral_secret",
  "expiresAt": 1786508008000,
  "model": "gpt-realtime-2.1",
  "callsUrl": "http://127.0.0.1:39343/v1/realtime/calls"
}

mint calls for two connect attempts: 1 (second reused the cached secret)
request path: /v1/realtime/client_secrets
authorization uses the standing key: true
standing key present in what the renderer receives: false
same secret reused: true

session sent to the API:
  audio.input.turn_detection: null
  audio.input.transcription.model: gpt-4o-mini-transcribe
  audio.output.voice: marin

Automated visual evidence

Download the deterministic macOS evidence · workflow run

Commit: 5f6857dfc83b6425478c7478b5c57940939ef3af
Scenario: smoke
Physical-notch check: not performed by CI

Physical-device evidence

Screenshot or screen recording: not attached
Physical-notch check: not performed
Device/display configuration: not recorded

Review follow-up

Bugbot found five real defects in the first commit; all five are fixed in 253c99c, each with a regression test.

Finding	Fix
Failed calls left the microphone held (high)	Every stop path tears down the device and peer before setting status
Overlapping `response.create` never cancelled (high)	User turns cancel the in-flight response; proactive sentences queue behind it
Voice honored `OPENAI_BASE_URL` against its own contract (medium)	Voice no longer reads it; the code now matches the README and the CSP
A transient `disconnected` ended the call (medium)	Only `failed` and `closed` are terminal — ICE passes through `disconnected` on a blip
The handshake could hang forever (medium)	One deadline covers the SDP exchange and the channel opening

A second pass found five more, fixed in 97869dc, again each with a regression test.

Finding	Fix
Session data never reached the conversation (medium)	The roster is pushed in on connect and on change, so the prompt no longer claims a capability the connection lacked
Interrupt raced the active response (high)	The interrupting turn waits for the cancellation to land; a newer interruption replaces an older waiting one
Channel close stranded the microphone (high)	An unexpected close tears down like any other stop
An error instead of `response.done` stuck the turn (medium)	An error frees the turn rather than queueing forever
Space release ignored while focus sat in the text field (medium)	Release ends the turn regardless of focus

A third pass found three more, fixed in 1a58778.

Finding	Fix
A failed call reported itself as "Voice off" (medium)	Teardown detaches the channel handler before closing, so the channel's own `onclose` cannot overwrite `failed` with `idle`
An already-fired deadline hung the handshake (medium)	`#waitForChannel` checks `deadline.aborted` rather than only listening for a future abort
A pointer slip off the talk button discarded the turn (medium)	The button captures the pointer, so releasing always sends; only Escape abandons

A fourth pass found two more, fixed in c29a7de.

Finding	Fix
Stopping mid-connect stranded the microphone (high)	Every early return out of `connect()` releases what it had already acquired
The panel showed "Audio active" while the microphone was closed (medium)	The waveform follows the developer only while listening and Luke only while responding

A fifth pass found one, fixed in 30565de.

Finding	Fix
An approved update arriving mid-press was lost (medium)	A held push-to-talk counts as a busy turn, so the sentence waits and is voiced after

It also had a second failure mode: speak() during a held turn started a response immediately, so Luke talked across the developer mid-sentence before the commit cancelled it. Interruption is now one-directional on purpose — the developer may cut across Luke, and Luke may not cut across the developer.

A sixth pass found two, fixed in b1462bc — both in the queue the fifth round introduced.

Finding	Fix
A queued sentence was dropped when the call ended (medium)	Teardown hands anything still waiting back, and the renderer captions it
Abandoning a press that overlapped Luke reported "Voice ready" (medium)	Status stays `responding` while a response is still running

Worth stating plainly: adding a queue to stop updates being dropped introduced a new way to drop them. Both fixes are covered by tests that fail without them.

Bugbot's seventh pass, against b1462bc, completed clean with no findings, and the Security Agent passed. The Approval Agent deliberately did not approve: this change crosses API-key minting, preload IPC trust boundaries, and microphone/WebRTC paths, so it asks for human review. That matches WORKFLOW.md — agents prepare evidence, humans own final review and merge.

A seventh pass, after the rebase onto #13, found two more — fixed in 533a6d2.

Finding	Fix
A typed Send during a held press talked into an open microphone (medium)	Requesting a response defers to a held press the same way proactive speech already did
The proactive-speech queue silently discarded its oldest entry (medium)	A sentence the queue evicts is handed back to be captioned

Both are the same mistake in two places: a guard applied to one path and not its sibling, and a bound that drops data on a code path that promises not to.

An eighth pass found two more — fixed in d6047c4.

Finding	Fix
A push-to-talk commit included however long the call had been idle (medium)	A turn starts by clearing the input buffer; a muted track still transmits and turn detection is off, so the server keeps everything since the last commit
A reply finishing mid-press released the queued turn (medium)	Ending the press is what flushes the queue, not the reply ending

Writing the second fix exposed a third defect it depended on: requesting a response during a held press set the status to responding, overwriting listening, so the turn stopped looking held — which is exactly what let the queue drain early. The status is now left alone.

A ninth pass found two more — fixed in a1cd398, both the sibling of a defect already fixed.

Finding	Fix
The peer's close handler survived teardown (medium)	The peer is detached before closing, like the channel already was — otherwise a failure's real reason is replaced by "The voice connection dropped"
A queued typed message was dropped on teardown (medium)	The pending turn carries its text, and an unsent one is returned to the draft field

Rather than wait for a tenth pass to find the next sibling, the rest of that shape is audited: every handler attached to the peer or channel is now detached in teardown except the channel's open and error handlers, which are deliberately left attached so a teardown racing the connect wait cannot strand that promise past its own deadline. Every piece of state teardown clears that a caller was told was accepted now has a handback.

Worth recording: the first regression test written for the peer handler passed against the broken code. An explicit close() sets the closed flag before tearing down, and that flag already guarded the handler; the defect only lives on the failure paths, which never set it. The tests now exercise those, and fail without the fix.

A tenth pass found two more — fixed in 7750dae.

Finding	Fix
A stop during minting was reported as a fault (medium)	Both exits from the mint ask whether the session was stopped before diagnosing anything
Space was claimed even with no call open (low)	Starting a turn reports whether it opened one, and the key is only claimed when it did — the panel body scrolls, so swallowing Space broke that

Two tests written for this round passed against the broken code before being retargeted. The first exercised a stop during a mint that succeeded, where a later guard catches it anyway; the defect needs a mint that comes back empty or throws. The second asserted the existing mid-connect stop still released the device, but the new guard makes that case abort before the device is acquired, so it now covers a stop during the SDP exchange — the point at which something is actually held.

Every fix in this PR is checked by reverting it and confirming the test fails. That check has now caught three tests that would have certified broken code, which is the main reason to trust the rest.

Finding counts per round: 5 → 5 → 3 → 2 → 1 → 2 → 0 → 2 → 2 → 2 → 2.

The second one matters more than its severity suggests: connect publishes the local stream with the track disabled, so the panel asserted that audio was being captured whenever voice was merely ready. Microphone state is the one thing this app must not misstate.

The first of these had a passing test that could not have caught it: the fake data channel's close() never fired onclose, so the re-entry a real channel guarantees never happened in the harness. The fake now behaves like the real API, and the regression asserts the status a tick after the failure. Reverting the fix makes the deadline regression hang rather than fail fast, which is the defect it is written to catch.

Notes

Rebased onto main through #19, which included the capsule/peek/panel restructure in #18. That deleted styles.css and reduced index.tsx to a bootstrap, so eleven of the fifteen commits here patched files that no longer exist. They are squashed into one so the feature ports once rather than eleven times; the review history is above.
The voice section renders below the tab bar rather than inside a tab. A caption arriving, or a turn being held, must not depend on which tab is open, and the capsule already carries the speech meter for both. Its caption list is height-capped so a growing transcript does not fight the content-sized panel.
The IPC layer merged without conflict: main.ts, contracts.ts, and preload.ts carry the realtime channels alongside #16's provider credential channels. app.tsx's own microphone handling is replaced by the voice session, which owns the track.
OpenAI is deliberately not added to CREDENTIAL_PROVIDERS. Those are the user's own keys for their own cloud sessions; inference is Luke's to provide, so asking a user to supply that key would be the wrong surface. It stays environment-configured for development. A shipped build cannot carry a standing key — it is extractable from a package — so production wants Luke's own backend minting the ephemeral secret. That is a one-file change: the renderer only ever receives a RealtimeConnection.
Rebased onto main after #14 and #15. Two conflicts were semantic rather than textual: LUKE-55 binds Escape to close its Settings view, and this branch binds it to abandon a held microphone. Escape now resolves in that order — abandon a turn first, then close settings, then collapse the panel — because leaving the microphone open is the worse outcome. LUKE-55's Settings view also replaces the session list inside the panel body, so the voice panel sits alongside it within the same scroll region.
LUKE-55 stores the Conductor key in encrypted in-app settings with an environment fallback, because a GUI launch inherits no shell environment. OPENAI_API_KEY has exactly that problem and is still environment-only here; moving it to the same store is worth doing, but it is a change to LUKE-55's settings surface rather than to this feature, so it is not folded in.
The WebRTC audio round trip is unverified. This Linux workspace has no microphone, no audio output, and cannot package the macOS app, so the live call — real getUserMedia, a real SDP exchange with OpenAI, and hearing Luke speak — has not been exercised. The state machine around it is covered by tests with injected browser pieces, and the mint path is covered end to end over real fetch, but someone needs to run ./scripts/run.sh on a Mac with OPENAI_API_KEY set and confirm the conversation actually works before this ships.
The voice panel changes the expanded panel's layout, so ./scripts/verify.sh screenshots need inspecting on macOS; the fixture path renders it with voice unavailable.

🤖 Generated with Claude Code

Open workspace in Conductor