feat(desktop): a call's Keyframe avatar ships as a bundle of its own, opened as a scoped session over the SDK's narrowest slice

Merged
CH
5d ago
+655-1

Plan item 6. desktop: Keyframe session bundle. Independent PR, base main.

What it adds

  • Dependencies. @keyframelabs/sdk@0.3.0 and livekit-client@2.22.4 are pinned exactly in apps/desktop devDependencies, beside the other bundled renderer libraries. 2.22.4 is the version the SDK's ^2.15.14 resolves to, and pnpm stands up one copy, so setLogLevel reaches the same logger the SDK's Room uses. The SDK's @google/genai peer is optional and is not installed.
  • The bundle. src/renderer/avatar-session/index.ts calls setLogLevel(LogLevel.warn) before any room exists. It then sets one global, window.lukeAvatarSession = { createSession }. It has its own esbuild entry in scripts/build.mjs, which outputs avatar-session.js beside call.html.
  • Budget. The new line is renderer/avatar-session.js: 148920 gzipped bytes, which is LiveKit's client plus the SDK. No other baseline moves, and renderer.js carries none of it.
  • Loading. call/avatar-session-bundle.ts loads it as avatarSessionModuleAtom = bundleAtom(...), following the whiteboard, plan-editor and file-tree precedent.
  • contract.ts. It holds the grant and the narrowest slice of PersonaSession and its options that the window drives, and imports no value. Effect stays on the renderer's side: the effect over the session is the renderer's, so the IIFE bundle carries no second copy of effect.

The contract (for #9b)

openAvatarSession(grant: AvatarGrant, module: AvatarSessionModule)
  : Effect<AvatarSession, AvatarSessionError, Scope>

interface AvatarSession {
  video:  Effect<MediaStreamTrack, AvatarSessionError>  // resolves on the SDK's onVideoTrack; fails CLOSED if the session closes first
  audio:  Effect<MediaStreamTrack, AvatarSessionError>
  sendAudio: (pcm: Uint8Array) => Effect<void>         // PCM16LE 24 kHz mono; this and the two below do nothing once closed
  endTurn:   Effect<void>                               // endAudioTurn()
  interrupt: Effect<void>                               // interrupt() only (lossy: lk.control, reliable:false); #9b's feed map does endTurn then interrupt
  speaking:  Stream<boolean>                            // false first, then each change from lk.playback_started/finished, deduped; ends on close; one reader
  closed:    Effect<AvatarCloseReason>
}
AVATAR_CLOSE_REASON = { USER, SERVER_CLOSED, ERROR }   // named as #1's avatar end reasons
  // client_initiated ā†’ USER; room_deleted (incl. the plan's duration cap) | server_shutdown | participant_removed ā†’ SERVER_CLOSED; unknown ā†’ ERROR
AVATAR_SESSION_FAILURE = { CONNECT, CLOSED }  // the error carries its kind only; #9b reports a CONNECT failure (a rejected connect()) as ERROR
  • Signature change from the pinned shape. openAvatarSession takes the loaded module as a second argument. The SDK lives in the bundle, so the caller hands over what avatarSessionModuleAtom resolved to. This is the same seam live-peer.ts gives the browser.
  • Scope. The session's close() is registered with the scope before connect(), so a connect that fails partway still leaves the room. Closing the scope is the one way this side ends a session.
  • Errors. No error carries the SDK's or LiveKit's message, because a connection error can name the room the grant opens.
  • Callbacks. They land in a Deferred or a Queue through doneUnsafe/offerUnsafe. There is no new Promise, no timer, and no Effect.run*.

Consumer: #9b (call window runs the session). Nothing calls openAvatarSession or avatarSessionModuleAtom yet. knip.json lists call/avatar-session-bundle.ts as an entry until #9b imports it, as #2443 did with the tile. The bundle's index.ts is a permanent entry.

Tests

avatar-session.test.ts runs against a fake PersonaSessionSlice at the SDK boundary:

  • the tracks resolve;
  • speaking follows the playback RPCs, once per change, and ends on close;
  • closed maps each SDK reason;
  • closing the scope leaves the room and gives CLIENT;
  • a close before a track fails the wait on that track;
  • a refused grant fails with CONNECT and no token, and still leaves the room;
  • feed calls reach the session, and a rejected write is not the caller's.

./scripts/check.sh exits 0.

Spike 0 findings (from code, not a live session)

The spike ran with no Keyframe key. Everything below comes from reading @keyframelabs/sdk@0.3.0 and livekit-client@2.22.4. None of it has been tested against a live Keyframe session.

  1. Pin. livekit-client is pinned at exactly 2.22.4, which is what the SDK's ^2.15.14 resolves to.
  2. Logging.
    • setLogLevel(LogLevel.warn) is enough for LiveKit.
    • The SDK's own console.* lines bypass it, but they carry no secret.
  3. Close report. closed resolves from onClose, not from our close() call. onClose('client_initiated') fires asynchronously after close() returns, and the fake reports it the same way.
  4. Reason mapping.
    • client_initiated maps to USER.
    • room_deleted maps to SERVER_CLOSED. This includes the plan's duration cap.
    • server_shutdown and participant_removed map to SERVER_CLOSED.
    • unknown maps to ERROR.
    • A rejected connect() fails with AvatarSessionError{CONNECT}, which #9b reports as ERROR.
  5. Close order. close() awaits every queued audio write, then calls room.disconnect() without awaiting it. The scope's finalizer awaits close().
  6. Interrupt is lossy. interrupt() goes on lk.control with reliable: false, so it can be lost. This is documented on AvatarSession.interrupt.
  7. No warnings after close. After a close, every SDK sendAudio warns on the console. sendAudio, endTurn and interrupt are therefore no-ops once closed has resolved, so a running feed doesn't warn 25 times a second.

Also untested live:

  • whether billing stops on close();
  • the order the tracks arrive in. The tests assume no order;
  • whether the playback RPCs fire as their names say.

Mac: CI could not verify this, and Mac evidence is pending. The bundle is built and budget-checked on Linux. Nothing here runs in a window until #9b, which also adds the connect-src for LiveKit to call.html.

šŸ¤– Generated with Claude Code

Open workspace in Conductor