feat(desktop): the call window runs the avatar's Keyframe session for as long as a grant stands, feeds it, and reports it (avatar 9b/12)

Merged
CH
5d ago
+675-31

Plan item 9b. call window: run the session. Stacks on #2476 (9a), #2471 (6) and the tile from #5, all merged into feature/avatar-video-keyframe.

What changes

  • Session (call/call-avatar.ts): a kept-alive rendererRuntime.atom turns main's grants (onAvatarSession, read through @sidecar/hosted/avatar-grant, not the barrel) into one openAvatarSession(grant, module) per grant, using a Stream.switchMap. The next grant, or nothing, closes that scope, and closing the scope closes the session. The same grant handed twice opens one session. The SDK bundle loads through #6's avatarSessionModuleAtom. speaking is not read: 9b has no consumer for it, and it allows only one reader.
  • Feed: onAvatarFeed maps exactly as Keyframe's bridge does: audio → sendAudio (base64 decoded), turn_end → endTurn, interrupt → endTurn then interrupt. A closed session ignores them (#6's whileOpen).
  • Report over ACT_KIND.AVATAR_REPORT, through a new non-hook tellAct in act.ts, which stays the only act caller:
    • STARTING once a grant stands.
    • LIVE on the video's first frame: the remote track's unmute, not just the track arriving.
    • ENDED with a reason: USER, SERVER_CLOSED or ERROR from the session's close. A CONNECT failure or a bundle that fails to load reports ERROR. A grant withdrawn under the session reports USER.
  • Mount: CallWindowCard draws CallVideoTile in place of the card while the status is STARTING or LIVE, the statuses 9a sizes the window for. The tile's <audio> is drawn in every state. The tile's local AVATAR_STATUS is gone and #1's is used instead. The tile gained an onAudioElement prop, so a scoped atom can retry a refused play() every second, the way use-voice-session.ts does.
  • Subscribed at the root: index.tsx mounts the avatar atom before the first state read. 9a's relay sends the standing grant on ready-to-show, which can come before React mounts, so a subscription made from a component could miss it. Panels and the voice window subscribe too, but no grant is ever sent to them.
  • CSP (call.html): connect-src wss://*.livekit.cloud https://*.livekit.cloud, with a comment. worker-src 'none' is kept, and media-src is unchanged.
  • Sentry scrub (sentry-breadcrumbs.ts): every renderer's Sentry.init now has a beforeBreadcrumb that drops (does not redact) any breadcrumb whose data.url or message contains access_token. That covers fetch, xhr and console. This fixes the leak the spike found: livekit's /rtc/v1/validate?access_token=… fetch on a failed join. It keeps the AGENTS.md carve-out true.
  • knip.json: removed the avatar-session-bundle.ts entry. AVATAR_SESSION_LOAD_FAILED is no longer exported. AvatarEndReason is now exported.

Tests

  • call-avatar.test.ts, with it.effect and a fake PersonaSession (#6's seam):
    • the feed maps to the right SDK calls, in order;
    • reports go STARTING → LIVE, and LIVE waits for the first frame, not the track;
    • each Keyframe close reason maps to its end reason;
    • a refused connect and an unloadable bundle each report ERROR;
    • a grant of nothing closes the session, reports ENDED USER once, and draws no tracks;
    • the same grant handed twice opens one session.
  • sentry-breadcrumbs.test.ts: the validate fetch, xhr and console breadcrumbs with a token are dropped, and a clean one is kept.
  • ./scripts/check.sh exits 0.

Untested against a live session

No live Keyframe session was ever opened. These are unverified:

  • the real server_url host behind the connect-src (#11's first real call confirms it);
  • that a remote video track's unmute lines up with the first drawn frame;
  • that the SDK's close reasons come in as #6 read them from its code;
  • that autoplay of the avatar's <audio> works in the call window without the retry.

Known gaps / for #11

  • Video off and Restart video on the tile do nothing yet: no act carries voice.setVideo until #11.
  • 9a shrinks the window back to the card on ENDED, so the tile's "Restart video" state is never drawn. #11 decides whether ENDED keeps the tile up.
  • The call window reports ENDED USER when main withdraws the grant. If main wants CALL_ENDED to stand when the call ends, #11 decides which write wins.

CI could not verify this; Mac evidence pending. A pull request builds nothing for the Mac, and this Linux VM cannot run ./scripts/verify.sh.

🤖 Generated with Claude Code

Open workspace in Conductor

Devin Review

Summary by cubic

Runs the avatar's Keyframe session in the call window for as long as main hands it a grant, feeds it Luke's voice, and reports it as starting, live, and ended.

  • Each grant opens one session that the next grant or nothing closes; the same grant handed twice opens one.
  • The feed maps as Keyframe's bridge does: audio → sendAudio, turn_end → endTurn, interrupt → endTurn then interrupt.
  • LIVE waits for the video's first frame, not the track's arrival, and ENDED carries the close reason, with refused connects and unloadable bundles reporting ERROR.
  • The call window draws CallVideoTile while starting or live and retries a refused play of the avatar's audio every second.
  • call.html now allows wss://*.livekit.cloud and https://*.livekit.cloud, and the avatar is subscribed at the renderer root so the standing grant is not missed.
  • No live session was ever opened, so the real server_url host, the unmute timing, and the SDK's close reasons are unverified.

Bug Fixes

  • Both the page's and the preload's Sentry now drop any breadcrumb that carries access_token, which LiveKit's failed-join validate fetch put in its URL.