fix(stt): retry refused Soniox transcription deletes so they cannot fill the account

Merged
KO
23m ago
+113-26

Problem

Soniox batch recovery (utils/stt/soniox_batch.py) deletes its temporary transcription after each call. If that DELETE is refused, the transcription leaks. That happens on a 429 under Soniox request-rate limiting, or when the transcription is still processing at the 120s deadline. Transcription ids are never persisted, so leaked ones are orphans, and they count toward Soniox's 2,000 stored-transcription cap.

Prod evidence (2026-11-15):

  • The Soniox account held exactly 2,000 completed recovery.wav transcriptions: 100 on Nov 8, 1,081 on Nov 10, 264 on Nov 12, 501 on Nov 13. The newest is 2026-11-13 11:10Z.
  • event=soniox_batch_cleanup outcome=failed per day matches those counts exactly.
  • Soniox create 429s: 0/hr through Nov 11, then ~420–510/hr from Nov 13 12:00Z on.
  • The Soniox breaker is open/half-open, and ~1,400 units are vendor_held{reason="rate_limited"}, cycling held → redriven → 429.

The 2,000 orphans were deleted operationally (2,000 Ɨ 204). This PR stops the leak from coming back.

Change

  • A refused transcription DELETE goes into a bounded per-process queue. The next calls retry up to 3 queued paths each.
  • The cleanup warning now carries http_status.

Tests

  • New sync test: a 429 on DELETE defers the path, and the next call deletes it.
  • New async test: a transcription still processing at the deadline (409 on DELETE) is deleted by a later call.
  • Both new tests fail on main. All 46 tests in the two Soniox batch files pass.

Failure-Class: none

šŸ¤– Generated with Claude Code