PR #11801 - BasedHardware/omi

Summary

Status note. #12181 (pause endpointing) has merged, so the first of the two always-on
surfaces is off this PR. Echo suppression stays, and is load-bearing rather than riding along —
c5f95d9 removes the wake word's own "assistant is speaking" guard precisely because
VoicePlaybackEchoPolicy.classify runs ahead of WakeWordService.observe.

Adds an opt-in Wake Word feature for macOS desktop: during ambient listening, say "Omi" followed by a command and the assistant runs it hands-free — no clicks, no Push-to-Talk.

What changed

Sources/WakeWord/WakeWordService.swift (new) — trigger engine: strips the wake phrase, extracts the command, and submits it to the assistant (openAIInputWithQuery). Guards: 30s cooldown against rapid repeats, per-segment-ID deduplication, user-speech only, and suppression while the assistant is already busy.
Sources/WakeWord/WakeWordSegmentParser.swift (new) — splits "Omi, do X" into wake phrase + command; requires a 2+ word command so a bare "Omi" never misfires.
AppState+ListenEvents.swift — feeds every incoming transcript segment into WakeWordService.observe(...), the single funnel for ambient speech.
AssistantSettings.swift — persisted wakeWordEnabled / wakeWordPhrase / wakeWordCooldown (default off so current users' behavior is unchanged).
SettingsContentView+General.swift — new Wake Word settings card with on/off toggle and a dynamic subtitle tied to the audio-recording mode ("Listens in the background during meetings and calls").
Tests — WakeWordServiceTests + WakeWordSegmentParserTests (15 cases).

How it works

ambient transcript segment ("Omi, order asian food")
  → WakeWordService.observe
  → WakeWordSegmentParser extracts wake phrase + command
  → gated: enabled? busy? user speech? cooldown? dedup?
  → onTrigger("order asian food") → submitted to assistant

Verification

xcrun swift test --package-path Desktop --filter WakeWord → 15 tests, 0 failures.
Exercised end-to-end in a running named bundle via the hermetic capture seam (drives the real handleBackendSegments → WakeWordService path):

WakeWord: submitting 'order asian food' to the assistant   ← wake phrase stripped
Transcript [ADD]: Omi, order asian food                    ← raw transcript line
Chat telemetry: chat_agent_query_started surface=floating_text
APIClient: POST https://api.omiapi.com/v2/desktop/messages ← real query submitted

Negative control: injecting "I was just saying the weather is nice today" produced no trigger (no wake phrase).

Changelog fragment added (changelog/unreleased/20260818-wake-word-trigger.json). Config rachets (check_desktop_test_quality.py, swift-format, SwiftLint) pass.

Follow-up (out of scope, separate issue)

While validating the demo bundle we hit a pre-existing macOS debug-build crash at startup in DesktopHomeView.restorePersistedCaptureServices → startTranscription (over-release, unaffected by this diff — reproduces with the wake word disabled and only manifests once mic permission is granted). Filed separately; does not block this feature.

Account isolation

A wake word fires under one account, but the realtime path can wait up to 5 s for its transport before it sends the command or falls back to chat. When the wake word fires, WakeWordService captures a RuntimeOwnerAuthorizationSnapshot; with no signed-in account it ignores the command. The snapshot is re-checked:

after the transport warm-up, in runWakeWordTurn;
before the command is sent to the realtime session (the in-progress turn is cancelled);
before the chat fallback;
on the default (non-realtime) chat route. The snapshot travels through submitHandsFreeCommand/submitSpokenCommand, sendFollowUpQuery, openAIInputWithQuery, routeQuery and the pending follow-up queue, and is checked again at the send itself (dispatchChatQuery and the direct sendAIQuery hop). A command queued behind an in-flight reply is dropped if the account has changed by the time it would be sent.

The route carries a HandsFreeRequest (the account snapshot plus a request id), and two small types in HandsFreeRequestScope.swift replace shared state the bar already had:

RequestBoundSlot replaces the optional pending follow-up. A queued hands-free query is handed out at most once, and only while its account is still current, including after a sign-out and sign-in as the same user.
QuietSurfaceLatch replaces the global "next query is quiet" flag. A request that is rejected before presenting releases only its own arm, so a rejected wake word can't hide the next typed answer, and an older request can never clear a newer one's.

After the awaited realtime send, WakeWordSendOutcome re-checks both the account and the active turn. A stale completion neither commits nor cancels a replacement turn.

An account switch therefore drops the command rather than sending account A's words as account B. WakeWordAccountIsolationTests (4) cover this:

A switch during warm-up is visible to the realtime path and blocks the fallback.
The same account still falls back to chat.
A command the realtime session accepted never also goes to chat.
The default check rejects a snapshot taken before a real account switch.
Both chat routes (hands-free and visible) and the realtime fallback receive the speaking account.

Removing the fallback check fails the first test. The two checks inside runWakeWordTurn consume the same tested closure, but have no test of their own, because that needs a live realtime session.

Screenshots

New — wake word enabled	Old — before (no wake word)

Tracked evidence: .agent-artifacts/ui-evidence/11801/001-desktop-macos-wake-word-enabled.png and .agent-artifacts/ui-evidence/11801/002-desktop-macos-before-wake-word.png.

Product invariants affected

INV-CHAT-1
INV-VOICE-1

Failure class (fixes)

Failure-Class: none

Line-count exception

openAIInputWithQuery and sendFollowUpQuery gained the turn-optional guard that lets a
wake word — which owns no VoiceTurnID — reach the visible surface as a voice query, so its
answer is spoken and a second command continues the conversation. Both are existing
functions in this file; splitting it is unrelated refactoring for a 16-line change.

Line-Count-Exception: desktop/macos/Desktop/Sources/FloatingControlBar/FloatingControlBarWindow.swift | 5785 -> 5885 | wake-word spoken answers and follow-up continuity route through the two existing dispatch entry points; the main sync (#13337 surface floor) and the quiet-answer latch fix add five more lines at the same dispatch and latch points The account snapshot a wake word captures is threaded through the same entry points, the pending follow-up queue, and both send checks, which is where queued dispatch happens.

Line-Count-Exception: desktop/macos/Desktop/Sources/FloatingControlBar/RealtimeHubController+SessionDelegate.swift | 1575 -> 1587 | Five lines recording session-spoken text into the playback history the echo policy reads; it belongs at the delegate that receives the text, and the surrounding turn-event guards are private to this file.

Line-Count-Exception: desktop/macos/Desktop/Sources/FloatingControlBar/RealtimeHubController.swift | 1644 -> 1751 | runWakeWordTurn shares runHeadlessPTTTurn's turn lifecycle (mint, route, open input, commit), so it sits beside it rather than duplicating those ordering rules in a sibling file. The added lines re-check the speaking account after each wait and after the awaited send, so a stale completion never commits or cancels a replacement turn.

Line-Count-Exception: desktop/macos/Desktop/Sources/AppState/AppState+Transcription.swift | 1785 -> 1786 | One argument line at the single beginRecording call site, telling the STT lane resolver whether the wake word needs a recognizer that can be told its name; the flag combination itself lives on AssistantSettings, not here.

Line-Count-Exception: desktop/macos/Desktop/Sources/FloatingControlBar/RealtimeHubSession.swift | 1821 -> 1836 | sendSpokenCommand is an eight-line sibling of sendTestTextInput sharing the same private buffering path; the wire form and buffer are private to this file.

Line-Count-Exception: desktop/macos/Desktop/Sources/FloatingControlBar/FloatingBarVoicePlaybackService.swift | 1449 -> 1511 | Echo recognition records what the two playback engines say out loud; the spoken-word history sits beside the AVAudioPlayer and AVSpeechSynthesizer speak paths that feed it, and both are private to this file.