feat(backend): name numbered speakers from transcript evidence
What
Names the speakers diarization leaves as bare indices, using a single pass over the finalized transcript, and only when a name is actually spoken.
detect_speaker_from_text catches a person only when they announce themselves in one of a fixed set of regex shapes. It misses "thanks, Alex" two turns later, and it cannot prefer a name the user already has a Person record for.
Why the model is not trusted directly
A wrong name shown once is an annoyance. A wrong name that enrols a voiceprint writes the wrong person's voice into that Person's speaker_embedding and mislabels them in every later conversation, silently, with no path back — #10453 shows how hard a merely lost embedding already is to recover.
So no single model output can cause a write. Two independent things have to agree.
1. The gate: grounding only
utils/speaker_resolution.py checks only what a machine can settle without reading the language:
- the quote appears verbatim inside one transcript segment (a quote stitched across two turns is rejected)
- that quote contains the claimed name
- the speaker index exists and is not already resolved by the user or by embedding matching
- the name is not the account owner's, and is not one of the #5223 stopwords
- two names on one speaker, or one name on two speakers, drops both
- confidence clears a floor
That is the whole gate. It deliberately does not judge whether the words mean the speaker is that person — "thanks, Alex" grounds identically whether Alex is the listener, someone being discussed, or a name read off a slide. Passing the gate is therefore not permission to write anything: ResolutionPlan has no assignments field, and everything it emits is a suggestion.
2. The refutation pass: what actually guards enrolment
Each surviving suggestion goes to a separate model call (utils/llm/speaker_verification.py) with the opposite job. It is shown one proposal, the transcript, and that speaker's own lines, and it is asked to break the identification — and to answer "refuted" whenever it is unsure. Only a suggestion the verifier fails to refute assigns segments and enrols a voiceprint.
Every failure mode resolves to refuted: a parse error, a timeout, an unreachable verifier, an unusable answer. A false refusal costs the user a tap on a suggestion; a false acceptance is unrecoverable, so the asymmetry runs that way on purpose.
An earlier revision of this PR did something else, and it was wrong. It asked the proposing model to also label its evidence (self_introduction / direct_address / …), auto-enrolled the top two labels, and used regexes to check the quote's grammar matched the label. That is circular — the label is just another output of the model being checked — and it pits hand-written patterns against a language model on the one question the model is better at. A self-audit found three ways through it. The taxonomy and the form checks are gone.
3. Reversibility, for when both are wrong anyway
An enrolment made from an inference is tagged llm_inferred — on the embedding and on the speech sample. Tagging only the embedding was not enough: SpeakerMatcher._recover_person_embedding rebuilds a missing embedding from samples[0], so clearing an inferred embedding meant the next listen session resurrected it from the same unvetted sample. Recovery now refuses inferred samples, and revoking clears both artifacts.
Reassigning a speaker away from a person whose enrolment came from an inference revokes it. Deliberately narrow: the person must be losing that speaker and hold no other segment in the conversation.
Also fixes speaker matching
The live matcher took the closest enrolled person inside SPEAKER_MATCH_THRESHOLD. As the roster grows the runner-up creeps inside the threshold too, and whichever of two similar voices scores marginally lower wins. select_best_match now requires the winner to beat the runner-up by SPEAKER_MATCH_MARGIN, and holds a lone candidate to a stricter distance, so a near-tie surfaces as an unlabelled speaker rather than a confidently wrong name.
Shape
| File | Role |
|---|---|
utils/speaker_resolution.py | the grounding gate — pure, no LLM, no I/O |
utils/llm/speaker_resolution.py | the proposal route (speaker_resolution, gpt-5-nano) |
utils/llm/speaker_verification.py | the refutation route (speaker_verification, gpt-5-nano) |
utils/conversations/speaker_resolution.py | side effects: Person, segments, enrolment |
utils/stt/speaker_embedding.py | select_best_match |
SPEAKER_NAME_STOPWORDS lives in utils/speaker_names.py so the gate can reuse the #5223 stopword list without importing av, numpy and Firestore. SPEAKER_IDENTIFICATION_PATTERNS, patterns_to_check and detect_speaker_from_text stay where they were, in utils/speaker_identification.py; nothing in the gate needs them.
Every Firestore call and every LLM call — including one refutation call per suggestion — goes through run_blocking(db_executor / llm_executor, …), so none of it runs on the event loop.
Acting on a suggestion
POST /v1/conversations/{id}/speaker-suggestions/{speaker_id}/accept— resolves the Person, assigns every segment of that speaker, drops the suggestion, enrols.DELETE /v1/conversations/{id}/speaker-suggestions/{speaker_id}— dismisses it.
Accept enrols as user_tagged, not llm_inferred. That is the point of the endpoint: an inferred sample is barred from rebuilding an embedding later, and a name the user tapped is a confirmation rather than a guess.
Resolve-or-create by name goes through get_or_create_person_by_name, which derives the document id from the normalized name so two conversations finishing at once contend for one record instead of coining two people for one human.
Rollout
Off by default behind SPEAKER_RESOLUTION_ENABLED. Runs once per conversation in post-processing, never in the live per-segment loop — #6901 was closed for adding weight to that path, and naming needs the whole conversation anyway. Enrolment is capped at 4 per conversation and refutation calls at 6, spent on the most confident suggestions first.
The pass runs as an async coordinator on its own loop rather than occupying a postprocess_executor thread across its awaits, is handed a deep copy so memory extraction is not reading segments it is mutating, and is scheduled after audio_files are durable because enrolment reads them back.
Tests
85 in test_speaker_resolution.py (grounding gate, refutation pass, write safety, usage attribution), 13 in test_speaker_label_suggestions.py (accept/dismiss/revoke), 10 in test_speaker_sample_attribution.py, 8 in test_person_identity.py (the create race), 8 in test_speaker_resolution_scheduling.py, 10 for the matching margin.
Notable ones: a refuted verdict assigns and enrols nothing; a verifier that raises degrades to suggestion-only; a failed segment write enrols nothing and leaves no orphan Person; a quote matching only inside a longer word ("I'm Annette" vs name "Ann") is rejected; an undiarized transcript resolves nothing; two concurrent resolvers of one unseen name yield one Person.
Review findings from Bugbot and Codex are actioned; see the comment thread. docs/doc/developer/backend/listen_pusher_pipeline.mdx is updated for the new pass and the real match-admission rule.
Refs #5565 items 6, 7 and 9.
Product invariants affected
None. Changes are confined to backend speaker identification and its post-processing entry point.
Failure-Class: none
No existing class covers "a ranked match accepted on an absolute threshold without requiring separation from the runner-up". Happy to add one if you think it is worth a definition.
[!NOTE]
High Risk
Touches Firestore transaction boundaries for transcript segments, person records, and voiceprint enrolment/revocation—errors or races could mislabel speakers or corrupt embeddings across conversations.
Overview
Adds speaker label suggestions end-to-end: conversations can carry proposed names for diarized speakers (segment id references only, no quoted transcript on the wire), and the Flutter client now deserializes and round-trips speaker_label_suggestions.
Post-finalization naming (wired in process_conversation via background scheduling) persists unverified proposals; accept and dismiss HTTP routes settle them with single Firestore transactions so concurrent accepts/dismisses cannot resurrect suggestions or overwrite segment assignments from stale snapshots. Manual and bulk speaker assignment clears matching suggestions and can revoke llm_inferred enrolments when a person no longer speaks in that conversation.
Person identity moves to get_or_create_person_by_name (deterministic id + transactional create) for idempotent people creation and accept flows. Speech samples and embeddings gain llm_inferred attribution: inferred samples are excluded from embedding recovery, and revocation clears both marked samples and inferred embeddings without touching user-confirmed voiceprints.
Live speaker matching switches to select_best_match (threshold plus margin / stricter lone-candidate rule) to avoid near-tie mislabels. Supporting changes include executor loop registration for cross-thread background tasks, redaction tests for suggestion fields, QoS entries for speaker_resolution / speaker_verification, and ratchet baseline updates.
Reviewed by Cursor Bugbot for commit a322930b627dbf6e6e0d0381728739fa9f78b986. Bugbot is set up for automated code reviews on this repo. Configure here.
Line-Count-Exception: backend/database/conversations.py | 1728 -> 2121 | speaker-label suggestion persistence, dismissed-speaker fence, and transactional segment assignment
Line-Count-Exception: backend/database/users.py | 2131 -> 2502 | speaker name identity keys, probes, and person-identity helpers
Line-Count-Exception: backend/routers/conversations.py | 1508 -> 1750 | speaker suggestion accept/dismiss routes and evidence surface
Line-Count-Exception: backend/utils/conversations/process_conversation.py | 1781 -> 1889 | finalization-time speaker resolution pass and suggestion persistence