Add Meta Muse as a transcription and intelligence provider

Merged
RE
2mo ago
+1241-49

Summary

Problem: A Meta Model API key only worked for Intelligence through the generic Custom entry. Transcription's Custom entry assumes a Deepgram-compatible /listen WebSocket, so Muse Voice Transcribe could not be used at all: Meta's ASR API is its own protocol (in-band auth handshake, raw PCM frames, typed turn events), not Deepgram- or OpenAI-shaped.

Fix: Adds a named Meta Muse provider on both tabs. Transcription gets a new owhisper-client adapter for Meta's realtime WebSocket and batch endpoint, wired through Provider, AdapterKind, BatchProvider, the listener and proxy dispatch sites, and the desktop catalog. Intelligence gets an OpenAI-compatible catalog entry that reuses the generic listing, key verification, and client paths, with muse-spark-1.3 ordered first.

What changed

  • Live transcription (crates/owhisper-client/src/adapter/meta/live.rs): connects to wss://api.meta.ai/v1/asr/realtime in DIARIZATION mode, sends the key in the handshake frame (the endpoint ignores HTTP headers), streams 16 kHz s16 PCM, and maps events to stream responses: cumulative transcript events become partials scoped to the open turn, speechComplete becomes the final, and speaker labels attach to the open turn. Selected languages are sent as languageBias (English names, 25 documented languages); keywords are forwarded.
  • Batch transcription (meta/batch.rs, behind the local feature): multipart POST /v1/asr/transcribe. Meta only accepts mono s16 WAV at 16/24 kHz, so segments are decoded and re-encoded with hound (new optional dependency in the local feature). Upload limit is 590 s per request under Meta's 10-minute cap.
  • Intelligence (apps/desktop/src/settings/ai/llm/shared.tsx): catalog entry with the existing Meta logo and fixed base URL https://api.meta.ai/v1. No lister case is needed: the generic keyword filter already drops muse-voice-transcribe-1.0 and muse-image-1.0, and one modelPriorityPatterns entry puts muse-spark-1.3 first so it becomes the default.
  • Desktop transcription settings: "Meta Muse" entry with the single model, shown as "Muse Voice Transcribe"; batch mapping and default model; regenerated plugins/transcription/js/bindings.gen.ts.
  • Provider suggestion registries: the live registry moves from the Tauri command into listener2-core beside the batch registry so both are plain functions with tests; both now include Meta.
  • Docs: one line each for the Transcription and Intelligence lists in docs/models-and-providers.mdx.

How the live event mapping works

Observed against the real API, the order at a turn boundary is speaker → speechEnd → speechComplete → next speechStart, and partials reset at each speechStart. The adapter keeps per-turn state (start_ms, end_ms, speaker index) keyed by turnId so speechComplete can arrive after the next turn has started. Meta returns turn-level timing only, so words are spread evenly across the turn, following the existing Mistral adapter.

Known limits

  • Partial transcripts carry the previous turn's speaker label until the turn completes; Meta emits the label at turn end.
  • Meta's 60-minute session cap closes the socket; the existing listener supervisor reconnect handles it, no adapter-level retry.
  • Direct BYOK only; no Anarlog Cloud proxy routing for Meta.

Verification

  • cargo test --locked -p owhisper-client --features local and without the feature
  • cargo test --locked -p listener-core, -p listener2-core --lib, -p transcribe-proxy --lib
  • cargo check --locked -p api, -p mobile-bridge, -p tauri-plugin-transcription
  • cargo clippy --locked -p owhisper-client --features local -p listener-core -p listener2-core -p transcribe-proxy -p tauri-plugin-transcription --all-targets --no-deps -- -D warnings (no new findings)
  • cargo test -p tauri-plugin-transcription export_types (bindings regenerated, diff limited to the BatchProvider union)
  • pnpm -F @anlg/ui build, pnpm -F @anlg/desktop typecheck, pnpm -F @anlg/desktop test, pnpm exec oxlint --quiet --format=github apps/desktop/src/, pnpm -F @anlg/provider-validation test
  • rustfmt --edition 2024 --check on changed Rust files; dprint check on changed TS/MD files; license-boundary scripts and node --test from ci.yaml
  • Live API with a Meta key: cargo test -p owhisper-client --features local meta -- --include-ignored (single and dual sockets, batch file), plus GET /v1/models and a muse-spark-1.3 chat completion
  • Manual on macOS with a local stable-config build: Meta Muse selected on both tabs, live transcription with speaker labels, re-transcribe after recording, Intelligence model list shows the five Spark models with 1.3 preselected, summary generation

[!NOTE]
Medium Risk
New third-party STT/LLM integration touches live capture, batch re-transcription, and API key handling; batch audio re-encoding and turn-level timing mapping could affect transcript quality or edge-case failures.

Overview
Adds Meta Muse as a first-class bring-your-own-key provider on Intelligence and Transcription, instead of relying on Custom endpoints that cannot speak Meta’s ASR protocol.

On the desktop, Intelligence gets a Meta Muse catalog entry (https://api.meta.ai/v1, API key) with branding via ProviderBrandImage, plus listing tests and a priority pattern so muse-spark-1.3 sorts first while voice/image Muse models stay filtered out. Transcription adds the same provider with muse-voice-transcribe-1.0 (“Muse Voice Transcribe”), default model wiring, batch provider mapping, and updated provider-order tests.

The Rust stack introduces MetaAdapter in owhisper-client: live WebSocket at /v1/asr/realtime (in-handshake auth, diarization mode, turn/speaker event mapping, interpolated word timing) and batch POST /v1/asr/transcribe (mono WAV re-encode via optional hound on the local feature, ~590s segment cap). AdapterKind / Provider / BatchProvider, listener adapters, direct batch routing, language suggestion (new suggest_providers_for_languages_live in listener2-core, Tauri command delegates there), and transcribe-proxy BYOK exclusions are all extended for meta.

Docs note Muse Spark for Intelligence and Muse Voice Transcribe for STT; plugin BatchProvider bindings include meta.

Reviewed by Cursor Bugbot for commit 1823255fdcde917e55372b3278c8b289fdbf9886. Bugbot is set up for automated code reviews on this repo. Configure here.