feat(avatar): Luke's greeting waits for his avatar to go live

Merged
CH
4d ago
+514-21

Why

In the first live test with Settings › Voice › Video calls on, Luke's first sentence played on direct audio while the Keyframe session was still connecting. #10's rule finishes a turn already playing on the element, so the avatar sat idle and silent through the greeting and moved only from the second reply on.

What

When video is on for the call, Luke's greeting (the cue that asks the model to begin) is held until the avatar is LIVE. If the avatar ends instead, the greeting is let go at once and the call goes on as voice. With video off, nothing changes.

The greeting is the server's: the voice function appends the opening on session.started and cues it once acknowledged. LIVE is the desktop's. So the gate crosses the wire, as agreed with the orchestrator (option C). The host decides the release, because it is the one place that knows every way the avatar can end.

  • Contract (@sidecar/features): session.create gains optional holdGreeting: true. A new device frame, session.greet {}, joins the report frames. JSON-schema goldens were re-recorded: create/opening gain the optional key, report gains the greet member, and there is one new golden for the greet frame.

  • Server (relay.ts, opening.ts, service.ts): on a created call that asked for it, the relay keeps the cue that answered the opening until session.greet. A greet sent before the acknowledgment lets the cue go as soon as the acknowledgment arrives. A hang-up or the device going drops it. A re-attach never holds.

  • Host (LiveSessionHolder): the voiceVideo setting is now read before session.create, so the service knows before session.started and there is no race. #letGreet sends session.greet once, on any of these:

    • avatarLive(), told while the avatar is granted;
    • #avatarEnded: unavailable, stopped, an error/server close from the avatar's window, #11's 20 s AVATAR_START_BOUND, or a start the session has no door for.

    A call that ends first sends nothing.

  • Main (AvatarLink): a LIVE report while the host holds a grant calls the new HostAvatar.live(). It doesn't depend on which window reports, so it holds when the parallel "video in the Call tab" PR moves the session to the panel.

No extra timeout, as asked. The ways into the hold are already bounded:

  • a start the service never answers ends at #11's 20 s bound;
  • a join that hangs after the grant fails on livekit-client's own connect and peer-connection timeouts. #6 maps that to CONNECT, which the window reports as ENDED/ERROR, and that releases the hold.

The #letGreet comment says so.

Known limits (from review):

  • A room that joins but never unmutes its video track is in neither case. It holds the greeting until the developer turns video off or speaks first. The call goes on; only Luke's opening waits.
  • LIVE is the first video frame (#9b). If Keyframe publishes its audio track noticeably later, the very start of the greeting could still miss the avatar's voice. This is a Mac retest item.
  • A device socket that drops and re-attaches during the hold loses the held cue, since the new connection has no opening to cue. A detach before the acknowledgment already lost it before this PR; the hold widens that window to the length of the Keyframe join.

Tests

  • apps/web/tests/voice-relay-greeting.test.ts (it.effect, in-memory sockets at the WebSocket boundary):
    • a call that held nothing is cued at once on the acknowledgment;
    • a held call is cued only on session.greet, and only once;
    • a greet sent before the acknowledgment lets the cue go when it arrives;
    • a hang-up drops the held cue.
  • voice-service-exchange.test.ts: holdGreeting on session.create reaches the relay end to end. Acknowledged and still held, the cue goes up after the desktop's greet.
  • packages/host/src/compose-live.test.ts (it.effect + TestClock):
    • with video on, create carries holdGreeting, and there is no greet before LIVE (a LIVE with no grant included) and exactly one after;
    • unavailable releases it;
    • the 20 s bound releases it, and not 1 ms before;
    • an error close before LIVE releases it, and the restart holds nothing;
    • with video off, nothing is held and no greet is sent, even with video turned on mid-call.
  • avatar-acts.test.ts: a LIVE report on a grant tells the host, and one with no grant doesn't.
  • I checked that the relay and service tests fail with the hold disabled.

./scripts/check.sh exits 0.

Evidence

CI could not verify this; Mac evidence pending. The user will retest on a Mac. With video on, Luke's first sentence should now lip-sync, and with video off the greeting should come as quickly as before.

Untested against a live session: no live Keyframe or GPT Live session was opened. Not yet seen against real services:

  • how long the greeting now waits on a real Keyframe join (the developer hears silence until the first video frame);
  • that a hung LiveKit join really fails through livekit-client's own timeouts and releases the hold.

🤖 Generated with Claude Code

Open workspace in Conductor