Voice: a dropped device socket detaches the call instead of ending it

Merged
CH
19d ago
+927-63

Why

In prod, voice calls end at about 300s (Vercel cuts the silent socket) and at 800s (the function's maximum duration). On the sessions route, the relay's onDeviceGone treated any end of the device's socket as a hang-up and sent session.close to OpenAI. That ended the WebRTC call even though the Mac re-attaches (reattachingSocket). In the logs the re-attach shows up as session-attached followed straight away by session-ended, because it lands on a session that is already closing.

The Mac's real hang-up already sends an explicit session.close over the socket (graceful-close.ts), and the relay forwards it. So a hang-up and a dropped socket can be told apart on the server.

What

  • relay.ts
    • The relay sets deviceAskedClose when the device forwards LIVE_CLIENT_EVENT.CLOSE, and sets refusedByPolicy in refuse.
    • When the device socket ends on VOICE_ROUTE.SESSIONS with neither flag set, the relay sends nothing upstream and settles at once with the new FINALIZATION.DETACHED (log.ts). The existing tail then closes only the sideband.
    • Everything else ends as it did before: the audio and introduction routes, refusals, real hang-ups, and a device that asked to close and then dropped.
  • LiveSessionService.release() does stop()'s bookkeeping (stops run events, clears the queue, tears down the standing session, writes pending rows, releases the transport) and sends no session.close. A stop() after it does nothing.
  • live-exchange.ts adds EXCHANGE_ENDING (CLOSE / DETACH) and HostedLiveExchange.endAs(ending). The scope finalizer calls release() on DETACH and stop() otherwise.
  • service.ts
    • The service passes the ending in before it closes the exchange scope: DETACH when the relay settled DETACHED.
    • On the still-starting path (the device left while the exchange was standing), it passes DETACH only for a sessions-route re-attach (opened.started). A fresh creation the device never heard an answer for is still closed.
  • A detached relay calls no record.close, so usage stays unconfirmed. The connection that later sees session.closed records the seconds, and the ledger dedupes them.
  • Drain: #drain now detaches sessions-route calls. I updated its comment, the relay and service header comments, and the README paragraphs on the service scope, the exchange finalizers, "One connection is one invocation", "How a session ends", and the writer's re-attach note.

The bound on a detached session

Security review flagged that, without a bound, a signed-in client could create a session, drop the socket, and keep its WebRTC up. Nothing would ever send session.close, so the session would run on Luke's key with no host watchdog. This PR adds the bound:

  • Migration 0052 adds a nullable detached_at to voice_sessions, plus a partial index over open, detached rows. voice-schema.ts declares both.
  • Stamp and clear. When the relay settles DETACHED, the service stamps detached_at on the row, but only while closed_at is null. The same stamp happens on the still-starting detach path. A successful re-attach (openAttached) clears it. The writes are new VoiceSessionRecord methods: detach, attached, detached, and closeLost.
  • The sweep lives in server/voice/orphan-sweep.ts and runs on /api/observation/tick right after the speech push.
    • It takes open rows stamped more than VOICE_DETACH_GRACE_MS (60s) ago, oldest first, at most 20 a tick, ten at a time.
    • For each row it attaches a fresh sideband through the existing LiveUpstream, sends session.close through the existing closeGracefully (now exported from @sidecar/voice/live-session), and on session.closed writes record.close and recordVoiceSeconds the way the service does. The seconds are recorded once through the ledger.
    • The attach and the wait for session.closed are each bounded at 5s. Worst case, the sweep takes two rounds of 10s out of the tick's 50s budget.
    • If the attach is refused (the session is already gone at OpenAI), or the session never answers, the row is closed as connection_lost and its last unconfirmed snapshot stands, so it is never swept again.
    • A deployment without OPENAI_API_KEY skips the sweep, the same kill switch as elsewhere. The tick's answer gains voice: { closed, lost, failed }.
  • Function externals: the tick now imports the voice upstream, so ws (bufferutil, utf-8-validate) and the Node built-ins it reaches are new externals of observation-tick.js, and also of default.js (the grouped dispatcher that mounts the observation routes). I regenerated server/function-externals.json with pnpm --filter @luke/web functions:externals. These are the same externals the voice functions already load.
  • README: "One connection is one invocation", "How a session ends", and "Scheduled Conductor observation" now describe the sweep.
  • For your approval: the CLAUDE.md scheduled-pass paragraph may want a sentence about this sweep, since it is the tick's one write that reaches OpenAI. I left CLAUDE.md untouched.

Tests

  • The hang-up helpers now send session.close before closing, through the new shared hangUpDevice in tests/support/voice-fakes.ts: hangUp, hangUpConnection, the two-connection loop, and the two hang-up tests in voice-service.test.ts. One assertion moved from framesToUpstream: 0 to 1, because the hang-up's own close is now that one frame.
  • Changed to the new behavior:
    • The service close/drain test now expects a detach: no CLOSE upstream, finalization: "detached", no record close.
    • The re-attach test's terminate() no longer expects a CLOSE.
    • The "seconds recorded once across a re-attach" test now sends the hang-up before terminate(), so it still checks the ledger dedupe.
  • New:
    • voice-service.test.ts: a terminated sessions-route socket sends no CLOSE, logs detached, and records nothing. A session.attach on a fresh connection resumes the session, and a later real hang-up records the seconds once.
    • voice-service-exchange.test.ts: a planning call dropped mid-call detaches, re-attaches, and its next ask (dl_2) lands in the plan's conversation.
    • live-session-service.test.ts: release() sends nothing upstream, writes the words said so far, and a stop() after it closes nothing.
    • voice-orphan-sweep.test.ts (new store test on PGlite, with the fake OpenAI): a detach stamps the row and a re-attach clears it. The sweep ends only the row detached past the grace and never re-attached, through session.close. It records the close and 42 seconds once, and a later sweep attaches nothing and records nothing more. A second test refuses the attach (404): the row is closed as connection_lost, its unconfirmed snapshot stands, no seconds are recorded, and it is not swept again.
    • The detach tests in voice-service.test.ts (fake record) and voice-service-exchange.test.ts (real record) now also check that the row is stamped on the drop and cleared on the re-attach.
    • hosted-observation-tick.test.ts checks that the tick runs the sweep at its own instant and carries voice in its answer.
  • With the relay.ts change reverted, the three new or changed relay tests fail.
  • The refusal, introduction, and audio-route tests are unchanged and pass.
  • ./scripts/check.sh exits 0.
  • test:store was not run here because this sandbox has no Postgres. CI's postgres job covers it, including the new voice-orphan-sweep.test.ts.

This is a server-only change. CI runs no Mac build and verify.sh was not run, but this PR changes no Mac code.

Known limits

  • Events in the 1–10s reconnect gap are lost on each drop.
  • A delegation in flight when the socket drops is lost, because the new connection gets a fresh exchange. Fixing this means resuming in-flight asks from the store on re-attach, or a make-before-break handoff.
  • The Mac-side ping that stops the 300s cut is a separate PR.

šŸ¤– Generated with Claude Code