feat(apify): one consolidated new-posts digest per scrape batch

Merged
SW
3mo ago
+488-38

Summary

The digest architecture for recoupable/chat#1855 (PR #4 of 6, decision: one email per scrape, all platforms): registers each platform run under a batch_id at scrape start, records completions + genuinely-new post URLs in the webhook layer, and sends one consolidated digest when the batch's last run completes.

Stacked on api#759 (newness diff — provides filterNewPostUrls); requires database#41 (batch columns) applied before this deploys. Base is #759's branch so the diff shows only this PR's work; GitHub retargets to main when #759 merges.

What changed

  • lib/supabase/apify_scraper_runs/ (new): insertApifyScraperRuns (start-time registration), completeApifyScraperRun (completion + new URLs), selectApifyScraperRunsByBatch, selectApifyScraperRun, local row type (generated types don't carry database#41's columns yet).
  • lib/apify/digest/ (new): maybeSendScrapeDigest (the brain: all-siblings-complete gate → per-platform sections → skip-if-nothing-new), getScrapeDigestRecipients (same watcher chain the solo alert used), sendScrapeDigestEmail (BCC-only, deterministic minimal body — upgraded to the house template in PR #5).
  • postArtistSocialsScrapeHandler: mints the batch, registers runs (with platform + social mapping via a new additive profileUrl on scrapeProfileUrlBatch results); response contract unchanged (extra field stripped).
  • apifyWebhookHandler: after each result handler, completes the run and runs the digest check; failures never fail the webhook. Schema now passes through the optional resource.id Apify already sends.
  • TikTok/Twitter handlers now compute + return newPostUrls (pre-persist diff); Instagram suppresses its solo email for batch-registered runs (legacy runs keep it).

Tests (RED→GREEN)

  • maybeSendScrapeDigest: 4 cases — waits for incomplete siblings; one send with per-platform sections (empty platforms omitted); nothing-new sends nothing; null batch no-ops.
  • Webhook: completion recorded with the handler's new URLs + digest check with the run's batch id; no-run-id skips bookkeeping.
  • Instagram: batch-registered run with new posts → no solo email, newPostUrls returned.
  • Sweeps: lib/apify + lib/artist + lib/socials → 272 tests passed; tsc --noEmit at exact baseline parity (200 pre-existing, 0 new); eslint clean.

Merge sequencing

database#41 → api#759 → this. E2E verification (real batch scrape → one digest) planned post-merge on the tracking issue since Apify webhooks call the deployed receiver.

🤖 Generated with Claude Code

Summary by cubic

Send one consolidated new-posts digest per scrape batch across all platforms. Implements chat#1855 by batching runs, diffing genuinely new URLs, and emailing a single BCC digest when the last run completes.

  • New Features

    • Adds apify_scraper_runs helpers: upsertApifyScraperRuns (batch registration), updateApifyScraperRun (completion + new URLs), selectApifyScraperRuns (optional { batchId }), and selectApifyScraperRun; helpers use generated Tables/TablesInsert types.
    • Webhook accepts optional resource.id, records completion with updateApifyScraperRun, and calls maybeSendScrapeDigest; digest errors never fail the webhook.
    • Digest module: getScrapeDigestRecipients, parseNewPostUrls (safe JSONB parser), maybeSendScrapeDigest, sendScrapeDigestEmail (per‑platform sections, BCC‑only; empty platforms omitted).
    • TikTok/Twitter handlers return newPostUrls (pre‑persist diff); Instagram returns newPostUrls and suppresses the solo email for batch runs (legacy runs unchanged).
    • postArtistSocialsScrapeHandler mints a batch_id, registers runs via upsertApifyScraperRuns with platform/social mapping (from profileUrl in scrapeProfileUrlBatch), and returns the same response shape (extra field stripped).

  • Migration

    • Ensure apify_scraper_runs has batch_id, completed_at, and new_post_urls before deploying; types/database.types.ts regenerated (database#47). No API response changes.

Summary by CodeRabbit

  • New Features

    • Added digest-based email notifications for newly scraped posts, including consolidated updates across platforms.
    • Scrape runs now track batch information and newly discovered post links for better follow-up handling.
    • Batch scrape results now include the original profile URL for easier reference.

  • Bug Fixes

    • Improved notification behavior so per-platform emails are suppressed during batch digest runs.
    • Webhook processing now continues successfully even if digest-related follow-up steps fail.