feat(catalog-value-estimator): seed deep historical backfill via the measurement-jobs resource

Merged
SW
4mo ago
+279-135

Why

Running the skill on a whole catalog left the Songstats backfill worker starved. Live prod DB (godremdqwajrwazhbrue, 2026-06-15): songstats_backfill_queue = 4 rows ever; songstats history for 1 song of ~4,290 Rostrum recordings; quota ledger 3 of 1,000 hits used. Root cause: enqueue is lazy (only a per-track historic-stats read enqueues, getTrackHistoricStatsApifyFirst.ts:36); the portfolio/snapshot path never calls it, so a 570-album capture seeds nothing. Queue starvation, not the quota exhaustion the baseline assumed. Tracked in chat#1796.

Design: REST resource model (not a new verb)

On KISS/REST review we dropped the originally-planned POST /research/backfill verb. The play-count surface is RPC-style and collapses to two resources (see chat#1796):

  • Reads → one measurements collection: GET /research/tracks/{id}/measurements with granularity (series, was historic-stats), aggregate=run_rate (was playcount-deltas — a projection), latest=true on /albums/{id}/measurements (was playcounts).
  • Writes → one measurement-jobs resource: POST /research/measurement-jobs {scope, source:"current"|"historical"} → 202 + pollable GET …/{id}. current absorbs snapshots; historical absorbs the dropped backfill. A historical job is the seed.

What changed (skill only)

  • estimate.py — portfolio (--album-ids) mode creates a historical measurement-job (POST /research/measurement-jobs {scope, source:"historical"}, ranked, deduped, free). Best-effort/non-fatal; --no-backfill-seed to opt out. provenance gains deep_history_share + backfill_seed; summary.md surfaces deep-history coverage.
  • recoup-api.md — documents the measurement-jobs ingest resource + a target-model mapping table (legacy per-track paths → measurements/measurement-jobs, marked deprecated aliases).
  • SKILL.md — explicit seed step in Step 2; corrected the false "auto-enqueued" History-depth claim; coverage note; resource-model pointer.
  • methodology.md — head-first prioritization + quota ceiling (~900 hits/30d), job-resource vocabulary.

Dependency / sequencing

Contract-first skill leg of a docs → api → skill sequence (chat#1796). measurement-jobs + measurements are not built yet; until they ship the seed degrades to a logged no-op and the reads keep calling the working legacy endpoints. The api migration keeps the legacy paths as deprecated aliases for one release, so nothing breaks at any point.

Testing

  • py_compile clean; --no-backfill-seed present.
  • seed_backfill smoke test on the job-resource shape — missing endpoint, error envelope, live (job_7 queued, enqueued:842), no-albums all return the right shape.
  • Skills gates green: portability_lint (64), check_vendored (153/35), validate_manifests (22) exit 0.

🤖 Generated with Claude Code

Summary by CodeRabbit

  • Documentation
    • Clarified how deep historical coverage is populated for portfolio mode, including when deep backfill is (and isn’t) automatically seeded.
    • Expanded methodology details on how measured_365d coverage grows over time and how related provenance fields evolve.
    • Updated API/backfill endpoint documentation, including request/response behavior and the move to the consolidated measurement-jobs approach.
  • New Features
    • Added automatic deep backfill seeding for portfolio runs to improve deep-history availability.
    • Added a CLI option to disable backfill seeding when needed.
    • Improved results with deep-history share and backfill seed provenance metadata.