feat: play-count write path — snapshots Workflow + cron backfill worker

Merged
SW
4mo ago
+1589-0

What

The write path of recoupable/chat#1791 — the final api slice. Fulfills the POST /api/research/snapshots contract (docs#238) and brings the backfill worker online against the claim RPC merged in recoupable/database#33.

  • POST /api/research/snapshots — exactly-one-of catalog_id / album_ids / isrcs (zod), resolution via existing catalog_songs + identifier mappings, per-org monthly cost cap (SNAPSHOT_MONTHLY_CAP_USD, default $25 → 429), job row minted in playcount_snapshots, workflow started fire-and-forget → 202 with estimated_cost_usd before any scraper spend ($0.003/album).
  • playcountSnapshotWorkflow — durable chunked capture (100 albums per actor call, each chunk an independently-retryable step), measurements written with run + snapshot lineage through a shared writeAlbumPlayCounts (which also DRYs the existing stats-refresh path). State machine on the job row: queued → running → done/failed.
  • songstatsBackfillWorkflow — the worker the architecture has been pointing at: budget = limit − reserve − rolling-30d ledger spend (ledger read failure = budget 0, fail-safe); claims value-ranked rows via the FOR UPDATE SKIP LOCKED RPC; one quota hit per track, recorded win or lose; historic points written as permanent songstats-labeled measurements (fetch-once — the read path's stitched historic serves them forever after).
  • Daily cron /api/internal/playcount-maintenance (mirrors the CRON_SECRET pattern via a shared validateCronRequest): starts the backfill drain and re-runs due monthly snapshot series (latest run per account+album-set older than 30d).

Workflow style mirrors the existing production workflows (buildOrgSnapshotWorkflow): thin "use workflow" orchestration, all logic in "use step" files, steps unit-tested directly.

Pricing note (decision surfaced, not made)

Snapshots are currently uncharged (no credit deduction) — the contract documents 202/400/401/429 with no 402, and abuse is cap-gated. A 570-album snapshot costs us ~$1.71; if snapshots should charge credits, it's a one-validator change.

Economics (from #1791's unit-economics section)

Every Songstats hit (~$0.09) spent here buys a permanent asset resold at ~pure margin by the read path — vs. the old request-path spend where the same 9¢ served one 5¢ call.

Tests

22 new tests across 11 units, every one RED→GREEN. Full suite 620 files pass; tsc −2 vs the test-branch baseline (202 → 200; the remaining are pre-existing); lint + format clean.

Verification plan (preview)

  1. POST /snapshots with album_ids: ["70Zkfb99ladZ3q0JVg97co"] → 202 + cost estimate → workflow run captures K.I.D.S. (Deluxe) → GET /playcounts shows fresh snapshot-lineaged captures.
  2. Cap check: tiny SNAPSHOT_MONTHLY_CAP_USD not settable per-preview — verified by unit test; live 429 deferred.
  3. Backfill: seed a queue row → hit the cron route with the bearer → drain runs → songstats-labeled history appears + ledger row + queue row done. (Songstats quota may still be 429-exhausted — if so, the failure path + ledger recording get verified instead, which is equally load-bearing.)
  4. 400/401/cron-401 paths.

Results table to follow as a comment.

Merge order

docs#238 ✅ → database#31/#32/#33 ✅ → api#657 ✅ → api#660 ✅ → this (closes the api surface; only the skill update remains on #1791).

🤖 Generated with Claude Code

Summary by cubic

Implements the play-count write path with a new snapshots API and a daily cron worker to backfill historical play counts, plus a type-safety fix in catalog song resolution. Adds durable workflows for snapshot capture and budget-gated Songstats backfill with clear lineage and cost controls.

  • New Features

    • POST /api/research/snapshots: accepts exactly one of catalog_id / album_ids / isrcs, resolves to Spotify album ids, enforces per-org monthly cap (SNAPSHOT_MONTHLY_CAP_USD, default $25 → 429), creates a playcount_snapshots job, starts the workflow, and returns 202 with snapshot_id and estimated_cost_usd ($0.003/album).
    • Snapshot workflow: chunked capture (100 albums per chunk), per-chunk retry, writes measurements with run and snapshot lineage via writeAlbumPlayCounts; job states: queued → running → done/failed.
    • GET /api/internal/playcount-maintenance: daily cron entrypoint gated by CRON_SECRET; starts the Songstats backfill and re-runs due monthly snapshot series (>30d since last).
    • Songstats backfill workflow: budget = SONGSTATS_QUOTA_LIMIT − SONGSTATS_QUOTA_RESERVE − rolling 30d ledger spend; claims rows via claim_songstats_backfill_rows (SKIP LOCKED), records one quota hit per track (win or lose), and writes permanent songstats-labeled measurements.
    • 22 new tests; types updated for the claim RPC; cron scheduled in vercel.json.
    • Type-safety: fixed param shape for selectCatalogSongsWithArtists; CI gate tightened to Next build type check.

  • Migration

    • Set CRON_SECRET for the internal cron route.
    • Optional tuning: SNAPSHOT_MONTHLY_CAP_USD (default 25), SONGSTATS_QUOTA_LIMIT (default 1000), SONGSTATS_QUOTA_RESERVE (default 100).
    • Vercel cron added: 0 7 * * * for /api/internal/playcount-maintenance.

Summary by CodeRabbit

  • New Features
    • Added playcount snapshot system to capture and track album play counts for research purposes.
    • Implemented Songstats backfill workflow to process historic track playcount data with quota tracking.
    • Added monthly snapshot scheduling for automated recurring data captures.
    • Introduced quota budget management to monitor API usage and spending limits.
    • Added cron-based maintenance handler for automated background workflows.