feat(voice): video calls with Luke's Keyframe avatar, behind Settings › Experimental

Merged
CH
3d ago
+18146-80

Video calls with Luke: a Keyframe Labs avatar on the GPT Live call, behind Settings › Experimental › Video calls (off by default).

This lands feature/avatar-video-keyframe as one squashed commit. The branch collected these PRs, each reviewed and green on its own:
#2467 (db: voice_avatar_sessions), #2469 (web: Keyframe client), #2470 (call video tile), #2471 (Keyframe session bundle), #2472 (contracts), #2474 (PCM tee and turn gate), #2476 (call window plumbing), #2478 (host frames and setVideo), #2537 (voice window feed path), #2545 (web: avatar frames and records), #2548 (run the session), #2563 (wire-up), #2565 (setting), #2734 (video at the top of the Call tab), #2737 (greeting hold, later removed in favour of main's #2623), #2902 (the Experimental toggle), plus a merge of main.

What it does

  • The voice service mints a Keyframe session on KEYFRAME_API_KEY and LUKE_KEYFRAME_PERSONA_ID, and records each one in voice_avatar_sessions. The sweep closes orphaned rows.
  • The voice window taps Luke's GPT Live audio, gates it into turns, and feeds it to the avatar, which runs in the panel and plays lip-synced audio and video at the top of the Call tab.
  • If the avatar fails, the call goes on as voice. "Restart video" asks again, and one unexpected close is retried automatically.

Secrets: the Keyframe session token reaches only the panel window (the AGENTS.md carve-out, approved). Sentry breadcrumbs and events are scrubbed of access_token.

Migration: 20261015203149_voice_avatar_sessions_keyframe, idempotent (IF NOT EXISTS, guarded FKs). It is safe whether or not production still has the table from the reverted LiveAvatar attempt.

Known limits (MVP, accepted): Luke's first sentence plays as plain audio while the avatar connects. A room that joins but never sends video holds the tile on connecting.

Verification: CI cannot verify the Mac. The developer tested on a Mac against this branch's Preview: video at the top of the Call tab, lip sync after the first reply. verify.sh evidence is owed. Before production use, set KEYFRAME_API_KEY and LUKE_KEYFRAME_PERSONA_ID in Vercel Production, and pick a Keyframe plan.

🤖 Generated with Claude Code

Open workspace in Conductor

Devin Review

Summary by cubic

Adds video calls with Luke's Keyframe avatar on GPT Live calls, behind Settings › Experimental › Video calls (off by default). The voice service mints a Keyframe session per call, records it, and the desktop panel runs the avatar: it tees Luke's voice into turns, lip-syncs them, and draws the video at the top of the Call tab. Video off collapses the tile to a compact strip whose camera button brings a fresh avatar back, and ending the call is the call bar's alone. If the avatar fails, the call keeps going as voice-only with a Restart video action.

Deployment

  • Migration 20261015231310_voice_avatar_sessions_keyframe is idempotent and safe against the reverted LiveAvatar attempt.
  • Set KEYFRAME_API_KEY and LUKE_KEYFRAME_PERSONA_ID in Vercel Production; without them a call asking for video goes on as voice.
  • The Keyframe session token reaches only the panel window and is scrubbed from Sentry breadcrumbs and events.
  • Luke's first sentence plays as plain audio while the avatar connects; a room that joins but never sends video holds the tile on connecting.

Refactors

  • Luke's voice is handed to the avatar only once the audio tee runs and the avatar's audio track has arrived, so the voice window never mutes him onto a dead avatar.