fix(desktop): end an on-device transcription window when the speaker pauses
Summary
On Apple Silicon, STTSessionState.resolveMode selects the on-device Parakeet path by default for microphone and system audio — cloud STT is the fallback, not the default. So ambient transcription runs through LocalTranscriptionService, which only closed a transcription window on a fixed 10-second boundary:
/// Window length transcribed at a time. Not real-time — gives a ~10 s "lag" like the user wants.
private let windowSeconds = 10.0
An utterance therefore waited for wherever it happened to land inside that window. Uniform arrival means ~5 s expected and ~10 s worst case, for speech that finished seconds earlier.
This closes a window when the speaker pauses instead. The 10 s cap is unchanged and still bounds continuous speech.
What changed
Three changes, all inside the on-device path:
- A window closes on a pause. Once the buffer holds ≥1 s of voiced audio followed by 0.6 s of quiet under the shared noise floor, transcribe it. The minimum is measured in voiced samples, not buffer length — a mostly-quiet window with one blip in it is exactly what Parakeet answers with a hallucinated word (a 1.1 s window at rms 0.0067 decoded to "Yeah." live).
- Leading silence is dropped, advancing the emitted-seconds cursor over it so absolute timestamps stay exact. Without this, quiet counts against the 10 s cap and the window fills partway through the next sentence.
- Pump tick 1 s → 0.5 s, since it is now the floor on how soon a finished utterance can be transcribed rather than a poll for a full window.
Measured
Real microphone, named dev bundle, on-device Parakeet, MacBook Air. Latency is speech-end wall clock to the transcription log line.
Before — fixed 10 s window, three identical utterances:
| utterance | latency | window |
|---|---|---|
| Omi, what time is it now | 1.08 s | [90.0s–100.0s] |
| Omi, what time is it now | 7.65 s | [120.0s–130.0s] |
| Omi, what time is it now | 6.18 s | [140.0s–150.0s] |
Identical speech, 1.08 s vs 7.65 s — the only variable is where the utterance fell inside the window. Every window is exactly 10.0 s wide and boundary-aligned, which is the tell.
After — 14 utterances across two builds:
| before | after | |
|---|---|---|
| mean | 4.97 s | 0.74 s |
| min | 1.08 s | 0.53 s |
| max | 7.65 s | 1.04 s |
| spread | 6.57 s | 0.51 s |
| theoretical worst case | 10.0 s | 1.2 s |
| window width emitted | 10.0 s fixed | 2.1–3.4 s, the utterance itself |
The spread collapsing from 6.6 s to 0.5 s is the part that matters: transcription arrives at the same moment every time instead of anywhere in a ten-second range.
Tests
LocalTranscriptionEndpointTests — 13 new cases pinning each half of the rule in both directions: speech-then-pause closes, speech-still-running does not, a pause shorter than the tail does not, pure silence never drains, a blip in a mostly-quiet window does not, voiced audio accumulates across short gaps, room tone under the noise floor reads as a pause, leading silence is trimmed with lead-in kept, a buffer opening with speech is not trimmed, and a pause between two utterances is never trimmed.
Verified on this branch, which contains no other change:
xcrun swift build -c debug --package-path Desktop— cleanLocalTranscriptionEndpointTests+LocalTranscriptionDuplicatePolicyTests— 18 tests, 0 failures
Why this is its own PR
Split out of #11801 (wake word) at the reviewer's request. Pause endpointing applies to every on-device transcription session, whether or not anyone enables the wake word, so it should not wait on a product decision about an opt-in toggle. It stands alone: this branch is main plus one commit, and the endpointing tests pass on it with no wake-word code present.
Honest gaps
- Parakeet still hallucinates a single word on quiet windows — twice in ~3 minutes of near-silence it emitted "Yeah." at rms 0.027–0.030, conf 0.60–0.61. Shorter windows surface these more often than 10 s windows did, so this change makes an existing behaviour more visible. I did not add a confidence floor: real speech in the same session ran conf 0.78–0.98 and the quiet-window hallucinations ran 0.35–0.79, so the distributions overlap and any threshold I picked would be fitted to one room.
- One machine, one room, one voice. 21 timed utterances is a clean before/after on a large effect, not a population.
- BLE/pendant capture is unaffected —
resolveModesends.bleDeviceto the cloud path, untouched here.
Failure-Class: none