PR #11804 - BasedHardware/omi

Summary

The desktop context-proactivity ("director") engine historically only evaluated a screen context on dwell — after you settle into a context through contextEntered, or right as you leave via evaluateAfterDeparture. Ambient speech was never a trigger: if you asked something aloud while a context visit was active, nothing happened until the next dwell.

This PR makes speech a first-class evaluation trigger. While a context visit is active, a fresh user utterance arriving through the ambient transcript funnel now evaluates that context immediately, grounded on the live screen plus a == LIVE SPEECH == section quoted from what you just said — so hands-free requests ("where does the deploy script live?", "what's open on this page?") get answered during ambient listening, not on the next dwell.

This stacks on top of #11801 (wake word); together the pair closes the loop: the wake word opens the mouthpiece, speech triggers the evaluation, and the director delivers a grounded answer.

Behavior

Before

Triggers were dwell-only: contextEntered and evaluateAfterDeparture.
evaluateAndDeliver produced a prompt composed of the tracked frame + task list + recent-delivery dedup only.
Ambient transcript segments built chat rows and nothing else.
No permission/eligibility round-trip was reachable without a dwell.

After

New entry point ContextProactivityEngine.evaluateFromSpeech(speech:) — reuses the exact same authorized-delivery tail as dwell evaluations (approve → gate → fence → admission-queue → eligibility → frame grounding), so behavior is consistent with dwell, scaled to one-shot speech slices.
AppState+ListenEvents feeds each new backend segment to SpeechProactivityCoordinator.shared.observe(...) (no-op when the feature flag is off).
The engine prompt now appends a bounded == LIVE SPEECH == section (up to 6 slices) behind the candidates section, so the director answers against both the screen and what you said.
Speech is treated as evidence, not instructions: slices are single-line flattened under an untrusted-speech boundary so nothing you say can forge prompt structure elsewhere.

Evaluation policy (admission)

A pure, unit-tested decision loop (SpeechProactivityAdmission) decides whether speech warrants an evaluation:

Check	Value
Feature flag	`isTranscriptProactivityEnabled` (off in prod/beta; on in dev bundles; `OMI_FORCE_SPEECH_PROACTIVITY=0` force-disables)
Conversation state	Must not be mid assistant voice-turn (uses `VoiceTurnCoordinator.activeTurnID`)
Speaker	Latest slice must be user speech
Minimum length	≥ 4 words (`minimumUserWordCount`)
Cooldown	≥ 90s since last speech evaluation (`evaluationCooldownSeconds`)

Skip reasons (admission.outcome) are observable: .flagDisabled, .conversationActive, .noUserSpeech, .utteranceTooShort, .coolingDown.

The speech window itself:

TranscriptSpeechSlice is a small Sendable value (segment ID, speaker, text, isUser, start/end).
SpeechProactivityWindow keeps max 12 slices, prunes by seen time (180s) — not backend timestamps (which can be in the future or missing) — and dedups by segment ID (transcripts are delivered repeatedly).

Files

Added

desktop/macos/Desktop/Sources/ProactiveAssistants/Core/TranscriptSpeechWindow.swift — slice + window types; SpeakerSegment.speechProactivitySlice extraction.
desktop/macos/Desktop/Sources/ProactiveAssistants/Core/SpeechProactivityAdmission.swift — pure admission policy.
desktop/macos/Desktop/Sources/ProactiveAssistants/Core/SpeechProactivityCoordinator.swift — @MainActor observer that hops to the engine actor.
desktop/macos/Desktop/Tests/TranscriptDrivenProactivityTests.swift — 14 hermetic tests.

Modified

Core/ContextProactivityEngine.swift — evaluateFromSpeech(speech:) entry; evaluateAndDeliver gained defaulted speechSection: appended after the candidates section. All existing dwell call sites compile unchanged.
Core/ContextBucketRollup.swift — ContextProactivityPromptBuilder.liveSpeechSection(_:maximumSliceCount:timeZone:); [You]/[Other speaker N] tags, ContextDestinationKey.singleLine(limit:) flattening, == LIVE SPEECH == header.
Core/ContextBucketsFeature.swift — isTranscriptProactivityEnabled + OMI_FORCE_SPEECH_PROACTIVITY override.
Core/ContextVisitCoordinator.swift — production activeFence() accessor beside the existing activeFenceForTesting().
AppState/AppState+ListenEvents.swift — funnel wiring after the new segment is built.

Notes / design choices

Serialization: speech evaluations ride the same dwellAdmission.begin(visitID:) queue as dwell evaluations, so concurrent speech + dwell evaluations can't interleave on the same visit.
Freshness: the same fence-freshness check applies; a speech evaluation won't fire on a stale fence.
Conversation-active uses VoiceTurnCoordinator.shared.activeTurnID != nil (WakeWordService isn't on this base yet; #11801 introduces it and this PR is stacked on it, so no behavioral conflict).

Testing / Verification

xcrun swift build -c debug --package-path Desktop — clean.
TranscriptDrivenProactivityTests — 14/14 pass (window dedup/retention/bounds, every admission outcome, prompt-section tags/flattening/slice cap).
ContextProactivity* engine suites — all pass (dwell paths unaffected by the defaulted param).
swift-format applied; swiftlint reports 0 violations.

Stack: #11801 → this PR.

Failure-Class: none

Line-Count-Exception: desktop/macos/Desktop/Sources/AppState/AppState+Transcription.swift | 1764 -> 1767 | Reset the transcript-driven speech window at the existing successful capture start, stop, and conversation-rotation lifecycle boundaries.