fix: attribute speaker budget blocks and schedule queued voices fairly

Merged
GI
7d ago
+1013-71

Repeated updates from a rejected voice can occupy FIFO work ahead of an untried owner and spend the last shared query token. This PR implements approved plan steps 1 and 2 together: exclusive budget attribution and fair admission through a coalescing metadata queue. It does not implement budget relaxation or fresh-audio cap resets (steps 3 and 4).

The production investigation could not distinguish the two caps or establish budget exhaustion as the primary cause of owner misses. The regression uses the real live runner: 80 updates from an eight-attempt rejector precede an untried owner with one socket token left; the owner is examined first, and only that request spends the remaining token. This is evidence of the scheduling mechanism, not a production impact estimate.

Metrics and interpretation

MetricMeaning / how to read
omi_speaker_id_budget_exits_total{cap=voice|socket|both}Every denied reservation has exactly one exclusive cap at the post-refill snapshot. voice: only 12-attempt cap; socket: only token bucket; both: both. Sum the three rates to equal the legacy omi_speaker_id_match_exits_total{reason="embedding_budget"} rate on the same instrumented targets.
omi_speaker_id_embedding_attempts_total{outcome=reserved|succeeded|failed}Reservations spend attempts/tokens synchronously before awaiting. Success means encoding/extraction returned, including later rejected/stale results; it is not identity acceptance. Failure includes encoding errors, extraction exceptions, and caller cancellation (executor work may still complete). Neither outcome refunds quota. Completions can lag reservations.
omi_speaker_id_budget_voices_total{cap=voice|socket|both}First denial per admitted numeric speaker per matcher generation, attributed to its first cap. Same numeric voice changing provider scope does not reset admission or incidence. Rollovers reset incidence even where carried attempts remain spent. Sum to count affected voice-generation cohorts, not unique users.
omi_speaker_id_budget_sockets_total{cap=voice|socket|both}First budget denial per socket lifetime, by its first cap. Includes voice-only denial and survives conversation rollover; this counts sockets affected by any cap, not only depleted socket buckets.

Compare exclusive exit rates for cap pressure, and summed exit/voice/socket incidence over a long window for repeat amplification. Coalescing can reduce repeated exit counts by itself, so fewer exits do not establish fewer owner misses; use the existing acceptance/contention and finalized-outcome signals alongside incidence. Incidence is counted when the first block occurs; it is not an active-exhaustion gauge. Equality with the legacy exit counter applies to new-version targets, so filter mixed-rollout comparisons consistently. For example: sum(rate(omi_speaker_id_budget_exits_total[1h])) versus sum(rate(omi_speaker_id_match_exits_total{reason="embedding_budget"}[1h])).

Each counter has three preinitialized closed-label values: 12 counter series per target plus 12 _created series. No user, socket, recording, conversation, generation, voice, or scope IDs are metric labels. One deduplicated speaker_id_budget_first_block metadata log per voice/generation carries UID, socket/recording/conversation IDs, generation, scoped speaker, spent attempts, refilled tokens, successful evidence seconds, pending seconds, and exclusive cap. Scope/cap changes do not repeat it; no transcript, audio, or embeddings are logged. Repeated legacy budget logs are suppressed while the legacy counter remains paired.

Fairness and cost bound

SpeakerMatchQueue is the shared scheduling primitive: at most 100 pending voices keyed by generation/conversation/scope/speaker, at most 128 merged distinct timing intervals per slot, latest segment ID, and metadata only. Repeated updates coalesce without renewing arrival age. Active dispatch is suppressed by numeric speaker ID across scope/generation changes, matching the unchanged matcher lock; pending keys remain distinct. After successful extraction, unselected already-queued speech remains in the same bounded queue for normal scheduling after completion; served work starts a new age, while already-pending updates keep their age and latest ID. A full queue leaves the ranges in matcher metadata for the next ordinary update. Failed requests and denials create no retries; every continued query reserves normally. Ready voices are ordered untried first, then existing acoustic near-owner preference, then longer pooled clips. At five seconds of age, oldest ready work overrides that preference; at most 99 older pending slots can precede it. This bounds selection starvation, not execution latency behind task capacity or active work.

Fairness cannot raise the GPU request ceiling: every extracted query still reserves through the same synchronous per-voice limit of 12 and shared socket burst of 16/refill of 240 per hour; failed/cancelled work still spends quota, and each query still uses at most 10 seconds of audio. No extra queue, timer, failed-request retry, quota exemption, or dedicated owner token exists. Continuing unconsumed successful work uses the same bounded queue and the same reservation gate. The socket envelope remains burst plus elapsed refill, with the existing rollover/carry rules; task concurrency stays at 16. Actual utilization/calls can rise within that ceiling when previously starved eligible work gets served. More pooled audio can affect per-call cost within the unchanged 10-second bound. Acceptance thresholds/margins, manual veto, contention arbitration, and stale publication fences retain their existing behavior.

Regression prevention and reviewer risks

Failure-Class: new

The new class is FC-live-speaker-budget-starvation. Root cause: per-update FIFO work can repeatedly schedule one rejector ahead of untried voices while a shared quota depletes; legacy repeated exits hide affected incidence and the denying cap. The durable prevention is a single bounded queue/admission primitive, backed by real-matcher tests for late-owner admission, age override, active-voice serialization, coalescing/gaps, queue bounds, fresh-evidence recovery, stale work, and concurrent failure ceilings. Telemetry tests prove exclusive attribution equals legacy exits, first-block dedup across cap/scope/rollover changes, closed cardinality, and reservation settlement for success/failure/encoding/cancellation/stale outcomes. Existing matcher suites retain threshold, manual-receipt, contention, recovery, reconnect, and rollover coverage. This is a shared primitive rather than a new bespoke source checker.

Product invariants affected: none (scripts/pr-preflight --lane local --suggest).

Reviewer risks: queue selection scans at most 100 pending items; age is a selection bound, not an SLA; incidence is generation-scoped and logs can recur after rollover; the latest segment ID represents coalesced work while the ordinary observed-segment inventory reconciles voice assignments; cancellation does not cancel already running executor/GPU work. No production reads or api.omi.me calls were made during implementation. No production outcome improvement is claimed before rollout measurement.

Validation

All backend tests use the repository runner with Google credentials/project unset and an isolated temporary Cloud SDK configuration. Target selection uses BACKEND_UNIT_TEST_FILE_LIST because backend/test.sh selects targets through that environment variable, not positional arguments.

  • Focused matcher/telemetry/queue/regression tests: 429 passed across 13 files via env -u GOOGLE_APPLICATION_CREDENTIALS -u GOOGLE_CLOUD_PROJECT CLOUDSDK_CONFIG=$(mktemp -d) BACKEND_UNIT_TEST_FILE_LIST=<focused-list> backend/test.sh.
  • env -u GOOGLE_APPLICATION_CREDENTIALS -u GOOGLE_CLOUD_PROJECT CLOUDSDK_CONFIG=$(mktemp -d) backend/scripts/run-unit-ci.sh --changed-files <list> --shard 4/3: passed all 431 selected files on the final revised code, including both new suites; Pyright reported zero errors. Earlier local announcement timing-only failures are retained in the evidence; this final run kept the same 1.0-second CPU guard and passed. The old profile-retry queue double is migrated to the real queue.
  • env -u GOOGLE_APPLICATION_CREDENTIALS -u GOOGLE_CLOUD_PROJECT CLOUDSDK_CONFIG=$(mktemp -d) backend/scripts/run-unit-ci.sh --changed-files <list> --shard 4/2: passed all 431 selected files after the teaching-persistence fixture correction; Pyright reported zero errors.
  • env -u GOOGLE_APPLICATION_CREDENTIALS -u GOOGLE_CLOUD_PROJECT CLOUDSDK_CONFIG=$(mktemp -d) OMI_PR_BODY_FILE=<draft> make preflight: all 32 checks passed on final head 2774019c17e387105032e46a3b40913a72dec9e7.

Publication retry uses the repository-supported PRE_PUSH_SKIP_PR_PREFLIGHT=1 hatch after the complete standalone preflight passed on the exact final head; the mandatory failure-class ratchet and every other push-hook check remain enabled. The first bounded hook passed, but its SSH connection closed before publication.

Automated review handling

The two queue findings are addressed with behavioral regressions: numeric active work prevents scope/generation duplicates from occupying lock-waiting task slots, and unselected clean intervals survive a noisy first query and recover through the same queue without another provider update. Continuations cannot exceed twelve voice attempts, budget denial stops continuation, and an untried owner can use the remaining socket credit. Full queues defer leftover metadata until ordinary stream activity; failures and denials never introduce automatic retries.

The identifier-sanitization suggestion conflicts with the explicitly approved UID/socket/recording/conversation metadata correlation requirement. sanitize() masks token-shaped UIDs/UUIDs and would prevent exact UID/session filtering. Raw authenticated/generated identifiers are intentionally retained only in the deduplicated first-block metadata log, consistent with the approved scope; never in metric labels and never alongside transcript, audio, or embeddings. Log access/retention remains a reviewer privacy consideration.

The teaching-persistence test helper now mocks only get on the real queue, preserving the retention API. Its sixteen cases pass in the focused runner; this fixes the one failing file reported by CI shard 2 without changing production behavior.