#673 fix(songstats-backfill): backoff on 429 + defer instead of churn (chat#1797 — pacing/backoff + logging)

Addresses chat#1797 bullet 1 (pace + bounded backoff on 429) and bullet 3 (per-step observability logging). Bullet 2 (remove the quota ledger) follows in a stacked PR.

The problem (live prod, full-catalog run)

The drain called Songstats faster than its per-second limit → ~79% 429s sustained for minutes, and backfillTrackStep recorded every 429 as a quota hit → the budget gate filled with phantom hits and would halt the drain with ~400 tracks still pending while the real quota was barely used. The reclaim sweep reset failed → pending and the workflow re-429'd them → a retry storm.

Fix

• fetchSongstatsWithBackoff (new): bounded exponential backoff on 429/408/5xx — retries the same request (~15s total, within a step's duration), honoring the rate limit; returns retriesExhausted when still rejected. 200 and non-retryable statuses (404) return immediately.
• backfillTrackStep: 200 → write/done/hit; 404/other 4xx → done/hit (terminal, real request); backoff exhausted → defer: leave the row pending, record no quota hit (Songstats consumed nothing), and signal the workflow to stop. Songstats is the rate authority — no phantom 429 spend.
• songstatsBackfillWorkflow: stop the run on the first deferral (Songstats saturated; remaining rows stay pending for the next drain) instead of churning failed → reclaim → 429.
• Observability: per-step (song, outcome) and per-batch (claimed N) logs so a drain is traceable in vercel logs (#1797 bullet 3).

Notes

• Rate adaptation is via the backoff (reactive throttling — a 429'd call waits and retries, naturally pacing to Songstats' capacity). An explicit token-bucket is a possible later refinement.
• The budget gate / quota ledger still exist here but no longer inflate from 429s — bullet 2 removes them entirely in the follow-up PR.

Testing

11 new/updated unit tests (backoff: 200/404/retry-then-success/exhausted/5xx/maxMs-cap; step: success/defer/no-data/terminal). Workflows + songstats + research suite 317 green; tsc/lint clean.

---

Summary by cubic

Adds bounded exponential backoff to the Songstats backfill and defers on persistent retryable errors without spending quota, preventing phantom spend and retry storms (chat#1797). Also simplifies queue updates with a single updateSongstatsBackfillQueue(ids[], fields) for single and bulk operations.

• Bug Fixes

  • fetchSongstatsWithBackoff: 4-step exponential backoff (~15s total: 1s, 2s, 4s, 8s cap); retries only 408/429/502–504; returns attempts and retriesExhausted.
  • backfillTrackStep: 200 → write/done/hit; 404/other 4xx → done/hit; backoff exhausted → keep pending, spend 0, return deferred.
  • songstatsBackfillWorkflow: stops on a defer and releases unprocessed claimed rows back to pending; adds per-batch and final summary logs.

• Refactors

  • Extracted retryability to isRetryableStatus and reused the existing delay helper; narrowed retryable 5xx to 502–504.
  • Collapsed queue helpers into updateSongstatsBackfillQueue(ids[], fields); used by releaseClaimedRowsStep and per-row callers (no-op on empty ids).

Summary by CodeRabbit

• Bug Fixes
  • Added bounded exponential backoff retries for transient failures (408/429/5xx) and surfaced retry attempt counts.
  • Improved rate-limit handling by deferring excess work and returning it to the queue for later processing.
  • Enhanced quota accounting and logging, including accurate spend reporting and clearer completion vs deferred outcomes.
  • Prevented partial drains from losing claimed rows by releasing them back to pending when processing stops early.