feat(research): measurement-jobs write resource + retryable backfill drain (chat#1796)

Merged
SW
4mo ago
+849-8

Implements the api consolidation for chat#1796 (write side), against the docs contract in recoupable/docs#242. TDD throughout — tests written first, RED confirmed, then implemented.

measurement-jobs — the ingest resource

Replaces POST /research/snapshots and the never-built backfill verb, and is what skills#43's catalog-value-estimator seed calls.

  • POST /api/research/measurement-jobs { scope, source }
    • source:"current" → reuses the existing snapshot pipeline (createSnapshot), maps snapshot_id → id.
    • source:"historical" → resolves the scope to ISRCs, enqueues each for Songstats backfill ranked by latest count, and skips songs already carrying songstats history (idempotent — no track fetched twice). Free. Returns { enqueued, skipped }.
  • GET /api/research/measurement-jobs/{id} — removed per docs#242 review (DRY): run status is covered by the generic GET /api/tasks/runs; the old snapshot flow had no status endpoint either. POST still returns the job id.

Layered per house style: route → handler → validate → data, one exported fn per file, reusing createSnapshot / selectSongMeasurements / upsertSongstatsBackfillQueue / resolveScope helpers.

Retryable backfill drain (robustness gap from #1796)

  • backfillTrackStep: a definitive 404 (no history) → done (terminal, never retried); transient 429/5xx/timeout → failed (reclaimable) instead of permanently stranding the track.
  • reclaimStaleBackfillRows: resets failed + orphaned in_progress rows to pending.
  • playcount-maintenance cron reclaims before each drain — so the 2 currently-stranded failed rows auto-recover on the next run.

Testing

  • 29 new/updated unit tests (validator, scope resolver, enqueue, dispatch, both handlers, status read, retry classification, reclaim, cron ordering) — all green.
  • Full lib/research + app/workflows suite: 291 passed, no regressions.
  • tsc --noEmit: clean on all touched files (pre-existing repo errors unrelated). Lint clean.

Scope / follow-up

This PR is the write consolidation + the drain fix. The read collection (GET /research/tracks|albums/{id}/measurements — repackaging the still-serving, now-deprecated historic-stats/playcount-deltas/playcounts reads) is the remaining api slice from #1796 and lands in a fast-follow. No endpoint is removed here — legacy paths keep serving.

🤖 Generated with Claude Code

Summary by cubic

Consolidates research ingest behind a new measurement-jobs write API for chat#1796, adds a Songstats card-on-file gate for historical jobs, and makes the backfill drain reliably retryable. Drops the bespoke status route; use GET /api/tasks/runs for status.

  • New Features

    • POST /api/research/measurement-jobs { scope, source, platforms? } (defaults platforms: ["spotify"])
      • source: "current" → reuses the snapshot pipeline and returns { id, state, album_count, estimated_cost_usd } (id mapped from snapshot_id).
      • source: "historical" → requires a card on file (402 + Stripe checkout URL when missing), resolves scope to ISRCs, enqueues Songstats backfill ranked by latest count in bounded batches (25), and skips songs that already have songstats history. Returns { enqueued, skipped }. Free.

  • Bug Fixes

    • backfillTrackStep: 404 and other permanent 4xx → mark done (terminal); 408/429/5xx → mark failed (reclaimable).
    • playcount-maintenance: calls reclaimStaleSongstatsBackfillRows before each drain to reset failed and stale in_progress rows (throws on DB error).

Summary by CodeRabbit

  • New Features

    • Added measurement job creation and status tracking endpoints
    • Support for historical data backfills and snapshot creation with flexible scoping (catalog, albums, or songs)

  • Bug Fixes

    • Improved handling of missing data scenarios from external sources

  • Chores

    • Added automatic cleanup of stale measurement jobs