feat: social_snapshots, posts engagement columns, apify_scraper_runs lineage (app#2018)

Merged
SW
2mo ago
+140-0

Database PR for recoupable/app#2018 (scrape persistence keystone). Contract: recoupable/docs#316. Absorbs the database row of app#2026.

Migrations

20260827120000_create_social_snapshots.sql

  • social_snapshots(id, social_id FK→socials CASCADE, captured_at, captured_on DATE, follower_count, following_count, post_count), UNIQUE (social_id, captured_on) so the api can upsert one point per social per UTC day (latest scrape that day wins), index (social_id, captured_at DESC) for the history read, RLS on with no policies.
  • Backfill: one row per socials row with a non-null followerCount, stamped with the row's updated_at, so history starts today rather than at the next scrape. ON CONFLICT DO NOTHING — re-runnable.

20260827120100_posts_engagement_columns.sql

  • posts.views / likes / comments / reposts as nullable BIGINT with >= 0 checks. Named columns rather than a JSONB so the table can be ordered and filtered on them.
  • Drops the set_updated_at BEFORE UPDATE trigger on posts. Every handler writes updated_at as the post's platform publish timestamp (and GET /api/artists/{id}/posts orders by it); the trigger would overwrite that with now() on the first re-scrape that refreshes engagement.

20260827120200_apify_scraper_runs_lineage.sql

  • origin TEXT CHECK IN ('artist','fan'), parent_run_id TEXT, account_id nullable (a spawned run inherits its parent's account; a chain whose root predates this migration has none to inherit). Partial indexes on parent_run_id and (account_id, created_at DESC) for the lineage walk and the budget PR's hourly count.

Merge order

docs#316 → this → api keystone (writes snapshots + engagement, registers spawned runs) → api budget. The api PR reads social_snapshots and the new posts columns, so this lands first; pnpm update-types in api regenerates database.types.ts after apply.

Verification

  • grep -c '\$\$' on all three files: 0 (no DO blocks, nothing for the shell to expand).
  • Backfill cardinality to be recorded on apply: SELECT count(*) FROM social_snapshots should equal SELECT count(*) FROM socials WHERE "followerCount" >= 0.

🤖 Generated with Claude Code

https://claude.ai/code/session_012PS8hmiwR1rGD6c41n6gD8

Summary by cubic

Adds the database layer for social snapshot history, post-level engagement, and scraper run lineage, so the API can persist scrape results instead of throwing them away after each run.

  • Creates social_snapshots keyed on (social_id, captured_on) for one follower-count point per social per UTC day, with a backfill so history starts today. captured_on is derived from captured_at by a trigger, so callers only supply the timestamp.
  • Adds views, likes, comments, and reposts columns to posts and drops the set_updated_at trigger that would overwrite the platform publish timestamp on re-scrape.
  • Adds origin, parent_run_id, and nullable account_id to apify_scraper_runs so spawned runs can be traced and counted.

Migration

  • Apply order: docs#316 → this PR → API keystone → API budget; run pnpm update-types in the API after applying.
  • Verification: the social_snapshots backfill count should equal the count of socials rows with "followerCount" >= 0.

Summary by CodeRabbit

  • New Features
    • Added historical daily snapshots for social follower counts, preserving values across future updates.
    • Added engagement metrics for posts, including views, likes, comments, and reposts.
    • Added scraper run lineage tracking to support related runs and origin details.
  • Data Integrity
    • Added validation to prevent negative follower, engagement, and activity counts.
    • Added automatic UTC-based dating for social snapshots.