Speech Profile Onboarding Fix & UI Improvements

Closed
NA
23d ago
+4203-1534

Summary

The Settings "Speech Profile" redo flow and real first-time onboarding share the same server-pushed question mechanism, but the backend's onboarding admission gate refuses any account with onboarding.completed=true — which is every real user reaching the redo feature from Settings. The socket connects and audio uploads, but no question ever arrives and Skip is a no-op, matching reports of "doesn't do anything" and "can't skip" on both the onboarding step and the Settings redo page.

  • Backend: add a speech_profile_redo flag that bypasses the completed-account admission check without granting onboarding provenance (no admission record, no conversation tagging). Real onboarding is untouched and still goes through the strict admission path.
  • Client: thread isOnboardingFlow through SpeechProfileProvider.initialise() so only real onboarding claims provenance; unify onboarding and Settings onto the single app-root provider instance (previously onboarding built its own second, never-initialized instance); fix a silent no-op on Skip when the socket is mid-reconnect; add a live mic-level meter (bar for onboarding, a reactive glow behind the device graphic for Settings); redesign the Settings speech-profile page (progress bar styling/animation, capsule buttons, title when a profile exists).
  • Pre-flight STT check: both entry points (onboarding's "Get Started" and Settings' "Get Started"/"Redo") now check STT availability first via a new endpoint backed by the existing per-process circuit breaker, and show an error dialog instead of ever entering a recording screen that can't do anything.

A pre-merge verification pass caught and fixed two real issues before this PR: an existing test's subclass override broke compilation after a new optional parameter was added (fixed), and unifying the provider instance meant onboarding needed to reset shared state on entry the same way the Settings page already does (fixed) — otherwise a prior account's leftover recording state could leak into a different account's onboarding in the same app session.

Follow-up fixes (same PR, added after the Modulate migration made Deepgram the 3rd fallback rather than primary):

  • The STT pre-flight check (/v3/speech-profile/stt-availability) hardcoded Deepgram's circuit breaker even after STT_SERVICE_MODELS made Modulate primary, so it reported on a provider the session would rarely serve from. is_stt_available() now resolves whichever provider is actually configured as primary (walking the same STT_SERVICE_MODELS list get_stt_service_for_language uses) and checks that provider's breaker instead — self-correcting across future provider reorders. Renamed the Dart client counterpart (isDeepgramAvailable → isSttAvailable) to match.
  • Brought onboarding's speech-profile step visually in line with the Settings redo page it shares a provider with: replaced the full-screen background photo with the same device-graphic + mic-level glow, added the "no device connected, using phone mic" disclaimer, restyled Skip as a white-outlined capsule, dropped the "Keep going" caption, hid the title once the profile completes instead of it reverting to "Please find a quiet place", and matched the transcript text styling (collapses to nothing when empty, same size/color as Settings). Deleted the now-unused bar-meter widget.
  • Unified the onboarding name step on one question ("What's your name?") regardless of whether a name was prefilled from sign-in, instead of switching to a different, smaller-font "Want to go by something else?" — removed the now-dead wantDifferentName string and its hasPrefilledName branch.

Test plan

  • dart analyze clean across the whole app (163 pre-existing baseline infos, 0 errors)
  • bash test.sh (Flutter): 1721 passed, 5 skipped, 0 failed
  • scripts/analyze_ratchet.sh: passed (several lint categories improved)
  • Backend: test_onboarding_question_start.py, test_listen_session_bootstrap.py, test_listen_runtime_regressions.py, test_listen_account_deletion_fence.py, test_streaming_deepgram_backoff.py (incl. 3 original + 5 new TestIsSttAvailable tests for the dynamic-primary pre-flight check) — all passed
  • Live end-to-end against a real Deepgram/OpenAI-backed local harness: an already-onboarded test account now receives real onboarding questions from Settings (previously impossible); progress/skip/reconnect behave correctly; new UI renders as designed; happy-path pre-flight check confirmed non-disruptive with real transcription in progress
  • Live end-to-end against the local dev harness (iOS Simulator, harness-seeded user signed in via Firebase custom token): confirmed the pre-flight check follows the real fallback chain (modulate -> deepgram in backend logs) rather than a fixed provider; walked onboarding through the redesigned speech-profile step and confirmed the device+glow, capsule Skip, no-device disclaimer, and collapsed-title/transcript behavior all render as designed; confirmed the name step shows one consistent question with a prefilled name

Product invariants affected: none

Failure-Class: new

Line-Count-Exception: backend/utils/stt/streaming.py | 1792 -> 1836 | Added _primary_streaming_service() alongside the existing is_stt_available() (formerly is_deepgram_available) so the client pre-flight check resolves against the actually-configured STT primary instead of a name hardcoded at the call site; no other change to this already-oversized file. Kept main's open_provider_selection_circuit that landed in the same region.

🤖 Generated with Claude Code

https://claude.ai/code/session_01FVsqiuLrNM7Jre5SyfATFC

Follow-up (2026-09-05): on-device STT fallback + transcript polish

Two more commits on this branch, prompted by live testing on an iPhone against the local dev harness:

  • On-device speech-recognition fallback for the speech-profile flow (onboarding step and Settings redo). The question flow only needs a transcript to drive questions/progress; the voice print is computed server-side from the uploaded WAV. So when the stt-availability pre-flight fails, or after the existing three 1011 closes with no captured speech (previously STT_UNAVAILABLE), the provider switches to the existing CompositeTranscriptionSocket with an on-device polling primary (Apple speech on iOS, downloaded Whisper on Android) forwarding suggested_transcript frames to the listen socket in custom_stt mode, which OnboardingHandler already consumes like server STT. No backend change; a receiver regression test pins that seam (test_onboarding_question_start.py).
  • iOS on-device recognition hardening (AppDelegate.swift): resolve the app's bare language code to an installed on-device locale (a recognizer built from "en" failed every request with kAFAssistantErrorDomain 1101); reply exactly once per clip (final result, error, or 20 s timeout) keeping partial results; and a real onDeviceAvailable probe on a silent clip so a phone with Siri/Dictation disabled (kLSRErrorDomain 201) is reported as "no local STT" up front and the pre-flight dialog tells the user to check the connection or turn on Dictation.
  • PurePollingSocket transcribe timeout (30 s): a provider that never answered left the processing flag set forever and silently stopped transcription for the session (found live). Frames are requeued and the next tick retries. Also protects the main app's on-device mode.
  • UI: subtler mic-level glow; live transcript now fades words in as they arrive via a shared FadeInWordsText widget on both screens.

Test plan (additional):

  • bash test.sh (Flutter): 1731 passed, 5 skipped, 0 failed; scripts/analyze_ratchet.sh passed
  • New tests: speech_profile_provider_test (5 fallback cases incl. one-way switch and close() reset), pure_polling_test (hung provider times out, requeues, next flush proceeds), fade_in_words_text_test (4), backend test_onboarding_question_start (suggested_transcript enqueued only in custom-STT mode)
  • Live on iPhone 16 Pro against the local harness with the STT primary forced unavailable: session connects with custom_stt+onboarding flags, Apple on-device recognition returns the spoken answer (~180 ms/clip) and it is forwarded to the backend; with Dictation disabled the probe reports unavailable and the dialog appears instead of a dead recording screen

Not changed: server-side behavior, provenance gating, or which recordings feed onboarding-source memory production.

Follow-up (2026-09-05, later): no Dictation dependency, "Answer with your voice" topics, sentence-based completion

  • iOS 26 SpeechAnalyzer for on-device speech (AppDelegate.swift). SFSpeechRecognizer's on-device mode fails with kLSRErrorDomain 201 whenever Siri & Dictation are off, which is what produced the "turn on Dictation" dialog. On iOS 26 the handler runs clips through SpeechAnalyzer/SpeechTranscriber (an app-installed AssetInventory model, no Dictation setting involved) and falls back to SFSpeechRecognizer only on older iOS or if the analyzer throws. onDeviceAvailable reports true once a supported locale's model is installed or installs within 8 s; a longer download continues in the background and the first transcribe() waits for it.
  • Speech-profile recording UX (onboarding step and Settings redo). No more one-question-at-a-time flow, percentage bar or "Skip this question": a compact white-outlined card headed "Answer with your voice:" lists three topics (where you live, what you do for work, your long-term goal) with a thin bar under it. The bar fills per spoken sentence (SpeechProfileProvider.targetSentenceCount = 3, counted on ./!/? boundaries followed by a space or end of text, Omi's own question segments excluded). Reaching it does not cut the user off: the provider waits 2 s without new speech (8 s cap) and then finalizes; the backend's onboarding_complete event no longer ends the recording. Onboarding keeps "Skip for now".
  • Live transcript: FadeInWordsText gained visibleLines; it replays the Wrap line breaking with the effective (theme-merged) text style and builds only the words on the last three lines, so earlier lines drop off whole and nothing is clipped or scrolled. Both screens show it in a bottom-anchored area with a three-line minimum height.
  • Settings page: Play plays the saved profile WAV in place through a playback-category audio session (loudspeaker, media volume) and turns into Stop; the samples page and its provider are deleted. No spinner while the page loads or the STT pre-flight runs. When recording ends, the page snapshots the recording view (last words, mic disclaimer, bar pinned at full, last mic level) and holds it still through the upload and a 1.5 s hold; then the words, card, bar and disclaimer fade out as one block while a plain white-bordered All done capsule fades in, and the mic glow eases shut over the same 450 ms instead of vanishing. The "'s Speech Profile" title scales to one line.
  • First-run onboarding no longer includes the speech-profile step or the memory-graph preview: Permissions goes straight to the completion screen; the speech profile is recorded from Settings. Both step widgets are deleted (their page indices remain placeholders).
  • Every recording starts empty: initialise() resets the transcript/progress first (Redo used to show and count the previous attempt's words).
  • Backend: ONBOARDING_QUESTIONS is the same three topics and OnboardingHandler._check_answer keeps the transcript across questions so one stretch of speech can cover several. LiveConversationController.prepare() always creates a fresh in-progress conversation for onboarding_mode sessions: within two minutes of a previous attempt the new socket used to attach to that attempt's still-open conversation and combine_segments() merged the new speech into its last segment, so the client replayed last time's words.
  • Dev harness: OMI_LOCAL_STORAGE_BASE_URL is advertised on OMI_DEV_HOST (new dev_advertise_host) so a phone can fetch local-storage links such as the saved speech profile.

Test plan (additional):

  • bash test.sh (Flutter): 1768 passed, 5 skipped, 0 failed; scripts/analyze_ratchet.sh passed; .github/scripts/check_dead_code.py ok
  • New/updated tests: speech_profile_provider_test (sentence target with grace/cap timing, no double finalize, Omi segments and backend completion ignored, transcript reset), fade_in_words_text_test (visibleLines keeps only the last whole lines), backend test_onboarding_talk_about_flow.py, test_listen_speech_profile_fresh_conversation.py, harness test_env_stage.py
  • Backend: listen/onboarding/recording-session suites (139 tests) pass locally; the full unit suite runs in CI
  • Live on iPhone (iOS 26.6.1, Dictation off) against the local harness: redo enters local-STT mode and transcribes without the Dictation dialog; Play fetches and plays the profile WAV on the loudspeaker and flips to Stop; card + bar render; transcript shows three whole lines; Redo starts empty; the recording finishes after the third sentence plus a pause and cross-fades to All done