fix(listen): recover a taught person's lost speaker embedding

Merged
AR
3mo ago
+146-6

What

A taught person can become permanently unmatchable, so the user sees only their own voice recognised — no matter how many times they teach anyone else.

Refs #10434.

Root cause

extract_speaker_samples stores the sample, then extracts the speaker embedding inside a try/except that only logs:

try:
    embedding = await run_blocking(sync_executor, extract_embedding_from_bytes, wav_bytes, "sample.wav")
    await run_blocking(db_executor, users_db.set_person_speaker_embedding, uid, person_id, embedding_list)
except Exception as emb_err:
    logger.error(f"Failed to extract/store speaker embedding: {emb_err} ...")

When that fails, the person keeps their (quality-verified) samples with no embedding, and nothing ever recomputes it. The listen loader requires a stored embedding, so it skips that person on every later session — and maybe_migrate_person_samples returns early for anyone already at v3, so re-teaching doesn't rescue them either.

The asymmetry that makes the symptom so lopsided: the user's own profile already self-heals from exactly this state — if its stored embedding is missing it re-extracts from the profile audio and writes it back (speakers.py, the has_speech_profile branch). People had no equivalent. Hence: own voice always recognised, everyone else never.

Fix

Give people the same fallback. When a person has samples but no stored embedding, rebuild it from their first stored sample and persist it.

Deliberately scoped so it does not reopen the quality gate. Recovery only runs for speech_samples_version >= 3. Those samples already passed verify_and_transcribe_sample when stored (the transcript that gates v3 is produced by that verification), so this restores a lost embedding rather than resurrecting a rejected one. The migrations intentionally drop embeddings for samples that fail quality, and that behaviour is untouched — which matters, because the failure mode of getting this wrong is misidentification, worse than non-identification.

Failures stay contained: one unreadable sample returns None and the remaining speakers still load.

Tests

backend/tests/unit/test_speaker_person_embedding_recovery.py:

  • recovers a v3 person whose embedding was lost, and persists it so the next session matches from storage instead of re-extracting
  • the recovered vector is shaped for compare_embeddings (matches under SPEAKER_MATCH_THRESHOLD), not merely non-None
  • a person with no samples is not recovered and storage is never touched
  • a storage/extraction failure is contained

Suites: test_speaker_id_pipeline.py + the new file → 112 passed.

Scope, honestly

This closes one deterministic path to the reported symptom — the one I could establish from the code. I can't see the reporter's account state, so I can't confirm it's the only contributor. A person can also stay unmatchable if migration fails repeatedly (maybe_migrate_person_samples returns the person unchanged on failure), which is a separate question about whether quality-rejected samples should ever be retried — a product call I've deliberately not made here.

Product invariants affected

none (verified with scripts/pr-preflight --suggest)

Failure-Class: none

No registered class covers "a derived artifact is lost by a swallowed exception and never rebuilt"; the durable guard is the recovery path plus its tests.