feat(hosted,desktop): Keyframe avatar frames, the avatar's status in app state, and the AvatarFeed reader

Merged
CH
5d ago
+639-10

Plan item 1. contracts of the Keyframe video-call plan. Independent PR off main. It adds the pinned shared interfaces and nothing that sends or reads them.

What lands

Frames (packages/hosted/src/live-contract.ts, exported from @sidecar/hosted):

  • Every avatar frame carries a required requestId. The host mints it (a UUID) for each avatar.start, and the other three frames echo it, so an answer or a stop left over from an earlier ask can be told apart from the current one. It is a bounded string, at most AVATAR_FRAME_BOUNDS.REQUEST_ID_CHARS = 64 chars.
  • Desktop → service: avatar.start {requestId} and avatar.stop {requestId, reason}. The reason comes from AVATAR_STOP_REASON = {USER, SERVER_CLOSED, ERROR}. A hang-up needs no stop of its own because the service closes the avatar along with the call. Read with avatarRequestFrameFromWire, which refuses any key the frame doesn't declare. For example, the desktop can't pick a persona.
  • Service → desktop: avatar.started {requestId, serverUrl, participantToken, agentIdentity}, read with avatarStartedFrameFromWire.
    • serverUrl must be an absolute wss: URL of at most 2,048 chars.
    • The token is at most 8,192 chars and the identity at most 256.
  • Service → desktop: avatar.unavailable {requestId, reason}, read with avatarUnavailableFrameFromWire. The reason comes from AVATAR_UNAVAILABLE_REASON = {NOT_CONFIGURED, NO_CAPACITY, ERROR}.
    • A reason from a newer service reads as ERROR rather than refusing the frame, the same way an answer drops a key it doesn't know.
    • A reason over 64 chars is refused.
  • The bounds are in AVATAR_FRAME_BOUNDS. The JSON Schema goldens for the five new schemas are recorded under LUKE_UPDATE_FIXTURES=1. All five are new files, so no existing golden changed.

Desktop vocabulary (apps/desktop/src/shared/messages/avatar.ts):

  • AVATAR_STATUS = {OFF, STARTING, LIVE, ENDED}.

  • AVATAR_END_REASON = {USER, SERVER_CLOSED, NO_CAPACITY, UNAVAILABLE, ERROR, CALL_ENDED}.

  • AvatarState {status, endReason?} and AVATAR_OFF.

  • state.voice.avatar is now required and set to AVATAR_OFF at launch.

    • A replaced voice window keeps it, through the new voiceWindowGone in app-state.ts, because the call window holds the avatar, not the voice window.

  • AVATAR_FEED_KIND = {AUDIO, TURN_END, INTERRUPT}. AvatarFeed is {kind:"audio", pcm}, {kind:"turn_end"} or {kind:"interrupt"}, read with avatarFeedFromWire. pcm must:

    • be non-empty, padded standard base64;
    • be at most AVATAR_FEED_BOUNDS.PCM_BASE64_CHARS = 2,560 chars, which is exactly 40 ms of PCM16LE 24 kHz mono (960 samples, 1,920 bytes);
    • decode to an even byte count, so it holds whole samples only.

    A shorter chunk is allowed so a turn's last chunk can be partial.

Consumers (no caller in this PR)

  • #7 (web: avatar frames and records): avatarRequestFrameFromWire, the started and unavailable frames, and both reason sets.
  • #8 (host: frames and setVideo): avatarStartFrameSchema and avatarStopFrameSchema for sending, avatarStartedFrameFromWire and avatarUnavailableFrameFromWire for reading.
  • #9a (main: call window plumbing): AVATAR_STATUS, AVATAR_END_REASON, AvatarState and state.voice.avatar for the AVATAR_REPORT act.
  • #10 (voice window: feed path): AvatarFeed, AVATAR_FEED_KIND and AVATAR_FEED_BOUNDS.
  • #11 (wire-up): avatarFeedFromWire for the VOICE_AVATAR_FEED act.

Tests

  • live-contract.test.ts:
    • a start that carries an extra key is refused, and so is a stop with an unknown or missing reason;
    • avatar.started is refused when the URL isn't wss:, a URL, token or identity is over its bound, the token is only whitespace, or the identity is missing;
    • an unknown avatar.unavailable reason reads as ERROR, and an oversized or non-string reason is refused;
    • a frame without its requestId is refused, the two answers included (they are lenient on the unavailable reason only), and so is a request id over 64 chars.
  • avatar.test.ts:
    • a full 40 ms chunk, a shorter chunk, a turn end and an interrupt are all accepted;
    • a chunk longer than 40 ms is refused;
    • each malformed feed is refused: empty, not base64, unpadded, half a sample, missing pcm, pcm given as an array, a turn_end that carries pcm, an unknown kind, and a value that isn't a record.
  • app-state.test.ts: the avatar starts OFF and is still standing after the voice window goes away.

./scripts/check.sh exits 0.

Mac

CI could not verify this, and Mac evidence is still pending. The PR touches main's app-state document and the voice-window-gone path in window-service.ts. Nothing is drawn differently.

🤖 Generated with Claude Code

Open workspace in Conductor

Devin Review

Summary by cubic

Adds the Keyframe avatar frames to the hosted live-contract vocabulary, the avatar's status to the desktop app state, and the AvatarFeed reader that carries the avatar's lip-sync audio. No callers yet; this is plan item 1 (contracts) of the Keyframe video-call plan.

  • New frames: avatar.start/avatar.stop {reason} from the desktop and avatar.started {serverUrl, participantToken, agentIdentity}/avatar.unavailable {reason} from the service, each string bounded; every frame echoes the request id the host mints for its avatar.start, so an answer or stop from an earlier ask is told apart; an unknown avatar.unavailable reason reads as ERROR, and a start carrying an undeclared key is refused.
  • state.voice.avatar is now required, starts as AVATAR_OFF, and survives voice-window replacement since the call window owns the avatar, through the new voiceWindowGone helper.
  • AvatarFeed accepts a padded-base64 PCM16LE chunk up to 40 ms, a turn end, or an interrupt; empty, unpadded, overlong, or non-whole-sample chunks are refused.