PR #24382 - BasedHardware/omi

What this does

Releases held audio automatically in two cases, both throttled:

Trial grant. Once per account, when a Stripe trial starts: the newest 25 hours of held audio.
Paid conversion. When a trial converts on a real charge, or a purchase on an active subscription raises the allowance: the newest held audio within the plan's remaining allowance.

Main already starts a conversion release, but only when the whole backlog fits the allowance. So the owners with the largest backlogs got nothing. Once the waiting-recordings cards (and their Process button) are removed in the separate UI PR, these releases become the path for held audio after an upgrade.

This PR is backend only. The branch's earlier app, l10n, analytics and changelog changes are reverted here (commit chore(app): leave the app UI out…); the UI cleanup ships as its own PR with renders.

David's ruling, in his words

2026-11-02 07:41Z: "Let's auto-release the last 100 hours of recordings automatically for newly trialed users. We need a way to throttle or prevent abuse for users with a large backlog, like let's say a free user builds up months of recordings and then upgrades. How should we handle it? On idle GPU capacity only?"
07:43Z, on the proposed limits: "Good with your prescriptions"
07:52Z, superseding 100h: "Let's keep on 25 hour trial grant, and lower other throttles as well for now"

The limits below scale the approved proposal by the same quarter. That scaling is the session's reading of "lower other throttles" and was reported to David.

Limits

Every automatic release passes utils/backlog_release.decide once per recording; a manual tap does not. A refusal leaves the recording held and the cursor unmoved, and the next sweep asks again.

Check	Limit
Kill switch	`BACKLOG_RELEASE_ENABLED`, on unless set to off. Off refuses every new recording and stops a release already in progress at its next recording (`processing_backlog_worker.run` returns before claiming). Audio already admitted to batch finishes; nothing is deleted.
Owner	5 h per UTC day, summaries included.
Batch route	Audio is released only when the owner is out of legacy sync. While legacy serves the owner, a hold would recover on the live Parakeet pool or a vendor, and the idle gate can't see either.
Normal priority	The first 2.5 h of a trial grant run at RECENT, the normal backfill class.
Idle (all other audio)	Runs at DEEP, only while fresh+recent ready < 64 and deep ready < 32. A missing or stale sample reads as not idle.
Fleet	20 audio-h/h and 400 audio-h per UTC day, shared Redis bucket, fails closed.
Grant	Newest first, once per account (ledger `trial_granted_at`). Older held audio stays locked; cancelling or lapsing the trial parks the release (`paid_owner` is re-read on every visit).

Held audio never takes the cheaper-vendor table (vendor_table=False).

Lead merge conditions

1. Measured idle signal. The HPA takes the larger of two signals:

ready units summed across all priorities, against a target of 32 per worker;
GPU-chain busy seconds from worker_flight, against a target of 0.5 per worker.

So a low priority alone does not make a release idle-only. Each signal is handled separately:

Ready units. The gate admits only while fresh+recent < 64 and deep < 32, so all ready units stay under 96. That is below the 128 that would add a fifth worker over the 4-GPU floor. A released recording adds about one unit (units are 3600 s).
Busy seconds. Flights where every item is DEEP now time as worker_flight_deep, which the adapter's worker_flight rule does not read. Released DEEP audio therefore cannot buy a GPU through this metric.

Measured on prod, prod-omi-backend, last 7 days, 10-minute steps, read-only GMP:

Measure	p50	p90	p99	max
Fresh ready units	0	111	503	599
Recent ready units	0	4	81	211
Deep ready units	0	0	0	6
GPU busy s per audio-s	0.048	0.073	0.089	0.17
GPU busy s/s while the gate reads idle	0.044	0.90	2.08	2.94
Worker pods while the gate reads idle	3	4	8	10

The gate read idle in 52% of samples. Samples with no ready metric (about a third of the window) count as not idle.
The last 6 hours held 4 pods.
Without the DEEP stage split, a full 20 audio-h/h release would add about 1.0–1.8 busy-s/s at the p50–p99 ratios. At the p90 idle baseline that would push the busy-seconds signal past the 4-worker floor, so the split is load-bearing.
DEEP ready never exceeded 6 in the week, so excluding DEEP-only flights barely changes today's scaling. Manual taps also run at DEEP and still scale through ready units.

2. Backfill tagged apart from fresh in hard-3 and the 2h alarm.

Each released recording carries release: trial|conversion on its intake record and on every batch job row.
Manual taps and fresh audio stay untagged.
Rows already carry priority. RECENT is shared with fresh-lane catch-up, which is why a separate tag is needed.
This PR changes no hard-3 threshold or reducer.
The collector change is Git-on-my-level/omi-workspace#44, for the lead to install with this release.
- A waiting tagged unit leaves T3, the 2h backstop (t3_pending_over7200) and fresh record latency for at most 24h, and is reported as release_t3_*.
- Terminal, T5, T1, T2 and T4 counts are unchanged.
- With no tagged rows the sample is byte-identical.

3. Daily cost against the $1,950/day target.

Ceiling: 400 audio-h/day × $0.13/h (rate card batch, 130 units per recorded hour, provisional, LLM and infra included) = $52/day.
Modulate worst case:
- Non-English held audio that Parakeet can't serve still goes to Modulate. That is $0.06/h card against Parakeet's $0.02/h.
- Those hours are reserved from the same 400 h fleet budget before admission, so they stay inside the cap.
- Worst case, every released hour on Modulate: +$16, for ≤ $68/day, about 3.5% of the target.
GPUs: idle-only DEEP work adds no marginal GPU spend on the 4-GPU floor. The 2.5 h normal-priority hours per trial can count toward the HPA, as approved.

4. Kill switch that stops in-progress releases. Covered by test_kill_switch_stops_a_release_already_running and test_kill_switch_stops_everything_including_priority_hours.

5. David's yes, in his words. Quoted above.

Known problems, stated rather than hidden

Earlier test run. The earlier backend run exited 0 with an empty output file, so it counts as not run. Reruns are below.
Modulate vs "no vendor overflow". Non-English audio still uses its provider route, because the GPUs cannot serve it. It is counted against the 400 h cap and priced above.
Batch is at 0. The batch-route check means nothing is released while legacy serves the owner, so nothing fires before the lead's re-entry deploy. No hard-3 behaviour changes.
Legacy-started holds. A hold whose legacy recovery had already started spending is passed over by automatic releases and stays held for a manual choice.

Luna round 1 (FIX-FIRST), both fixed

P1, trial start released more than 25h. A Basic owner starting an Unlimited trial raises the allowance, so the conversion path began an unlimited release before the trial grant arrived.
- Fix: a conversion now releases only when the Stripe subscription is active. That covers the subscription event, checkout and immediate plan change, all now in processing_backlog_worker.
- A trialing subscription is covered by its grant and converts later on trialing→active.
- An unknown Stripe status waits.
P2, legacy-started holds escaped the controls. A hold whose legacy recovery already spent can only finish on the legacy route, outside the idle gate and the vendor table.
- Fix: an automatic release passes over it (decision legacy_started) like audio that does not fit, and it stays held for a manual choice.

Luna round 2 (delta, FIX-FIRST), both addressed

P1, a stale legacy-spend check, and resumed reservations skipped it.
- Fix: _process rereads the hold before every automatic attempt, resumed reservations included, and defers legacy-started audio (it stays held).
- Backstop: recover_selected refuses the legacy fallback whenever an automatic release is in scope, after its own row claim. The next visit rereads and passes over the hold, so nothing stalls.
P2, a transient Stripe lookup failure silently skipped a release.
- Fix: an immediate plan change passes the modified subscription's own status (from stripe.Subscription.modify), so it does no second lookup.
- Checkout retries the lookup three times and counts a persistent failure as omi_processing_backlog_choice{source=conversion,outcome=error}.
- Retry for that case: a checkout whose Stripe lookup fails three times is still released by the customer.subscription.created event for the same purchase. That event carries the subscription's status in its payload, so it needs no lookup, and a paid subscription created active releases as a conversion. Creation happens once per subscription, so a renewal never starts a release. The lookup failure stays counted. An unknown status is never treated as a paid charge, which would reopen P1. (The lead noted the UI PR #24397 removes the manual Process button, so this event path is the retry.)

At the two-round cap, the lead (via uitrial, 2026-11-02 15:20Z) accepted this as SHIP with no third review. The conditions were exact-head CI green, and this body stating the Modulate hours inside the 400 h cap and the r2 residual. The event-path retry above closes the lead's follow-up on that residual.

Verification

Targeted: BACKEND_UNIT_TEST_FILE_LIST=<changed and related files> bash backend/test.sh (18 release files, then 18 payment and billing files after the router refactor, all passed). The first run had 37 passed and 1 failed: test_fleet_snapshot_runs_beside_the_sweep_not_ahead_of_its_work, a 2-second thread-timing check under host load, which passed on rerun along with the fixed gate test (38 + 26 passed).
Full backend unit suite: see the comment below.
Prod measurements: GMP PromQL under the ro-prod tier, aggregates only.

Deploy needed (lead)

Backend images that run the processing-backlog worker and payment webhooks (backend), shared batch intake (backend, backend-listen), and batch-pipeline-worker (the worker_flight_deep stage). New Redis keys: rl:backlog-release:v1:*. New metric: omi_held_content_backlog_release_total{decision}. Rollback: BACKLOG_RELEASE_ENABLED=false stops all automatic releases at their next recording.

Product invariants affected

none

Failure class (fixes)

Failure-Class: none

🤖 Generated with Claude Code