feat(apify): deterministic digest template — direct links to each new post, no LLM body

Merged
SW
3mo ago
+849-73

Summary

The content half of the revamp on recoupable/chat#1855 (PR #5 of 6, stacked on api#760): a shared deterministic renderer replaces the per-send LLM body. Evidence on the issue showed the LLM body produced different branding on every send, leaked vendor jargon ("Apify Dataset Notification") to customers, and didn't reliably link the posts it claimed were new.

What changed

  • lib/apify/digest/renderScrapeDigestHtml.ts (new): deterministic subject + HTML — per-platform sections, a direct link to every genuinely-new post, friendly platform labels, chat CTA as secondary link. No LLM call.
  • sendScrapeDigestEmail (batch digest) and sendApifyWebhookEmail (legacy solo alert) both use it. The solo alert now takes newPostUrls and sends nothing without them; its LLM path (generateText) is deleted.
  • Note on api#758 overlap: this branch rewrites sendApifyWebhookEmail.ts wholesale and embodies the same BCC invariant (tests recreated here) — merging #758 first then this stack resolves in favor of this version with no behavior change.

Tests (RED→GREEN)

  • Renderer: links every post grouped by platform; byte-identical output for identical input; no "apify"/"dataset" wording anywhere customer-visible; subject counts posts and platforms.
  • Solo alert: BCC invariant (×2), links present, no-recipients and no-new-posts short-circuits.
  • Sweep: vitest run lib/apify all green; tsc at baseline parity; eslint clean.

🤖 Generated with Claude Code

Summary by cubic

Replaced the LLM email with a deterministic, house‑style digest that links to every genuinely new post with media, captions, UTC dates, and compact stats; emails are addressed by the artist name, show the Recoup logo, include a roster footer, and the CTA always points to the chat app. Also adds X/Twitter post extraction and a LinkedIn label, ignores retweets and replies to avoid misattributing engagement, and fetches 10 tweets by default so original posts surface after filtering.

  • Refactors

    • Added focused helpers and extractors, including extractTwitterPosts, extractArtistNameFromDatasetItems, formatPostStats, formatUtcDateLabel, getPlatformLabel, and escapeHtml; platform labels cover X/Twitter and LinkedIn.
    • Added getRunDigestSection and updated maybeSendScrapeDigest to re-read run datasets for enrichment, degrade to URL‑only on failure, address the digest by the profile’s display name, and default X/Twitter scraping to 10 items to survive the retweet/reply filter.

  • Migration

    • Update calls to sendApifyWebhookEmail(profile, emails, newPostUrls); newPostUrls is now required.
    • No email is sent (returns null) when emails is empty or newPostUrls is empty; recipients are BCC‑only and to is set to RECOUP_FROM_EMAIL.

Summary by CodeRabbit

  • New Features

    • Added consolidated scrape digest emails for newly discovered Instagram and TikTok posts.
    • Digest emails now include platform sections, artist names, captions, thumbnails, timestamps, and engagement metrics.
    • Added clear subject lines, compact metric formatting, platform labels, and links to the website and chat experience.
    • Posts remain visible as URL-only entries when enrichment data is unavailable.

  • Bug Fixes

    • Corrected webhook notifications so newly discovered post URLs are included in outgoing emails.
    • Improved handling of missing or invalid scrape data without preventing notifications.