feat(apify): scrape alerts fire only on genuinely new posts (diff against stored posts)

Merged
SW
3mo ago
+95-5

Summary

Kills the alert false positives: the notifier sent "new posts" whenever a scrape returned any posts — no diff against what's stored — so every scrape re-announced the profile's recent feed. Evidence on recoupable/chat#1855: 6 of 7 alerts to one customer in one day announced posts that were 2–10 days old and already in the database.

Tracking issue: recoupable/chat#1855 (PR #3 of 6 — independent; biggest notification-volume win).

What changed

  • lib/socials/filterNewPostUrls.ts (new, reusable across platforms): given candidate post URLs, returns those with no existing posts row. Documented constraint: must run before the upsert (afterwards nothing is distinguishable as new).
  • lib/apify/instagram/handleInstagramProfileScraperResults.ts: computes the diff pre-upsert and only sends the alert when ≥1 URL is genuinely new. Persistence is unaffected — posts/socials/social_posts/follow-up scrapes run exactly as before; only the notification is gated.

Tests (RED→GREEN)

  • filterNewPostUrls: known-vs-new partition, all-new passthrough, empty-input short-circuit (no query).
  • Handler: new case — all posts already stored → no email, everything else persists; existing happy-path (new posts → email) unchanged.
  • Suites: vitest run lib/socials lib/apify/instagram → 13 files, 63 tests passed; eslint clean.

Relationship to the digest PRs

The digest aggregation (PR #4) consumes this same helper per platform. This PR is deliberately shippable alone for immediate relief on the current per-platform email.

🤖 Generated with Claude Code

Summary by cubic

Stop Instagram scrape alerts from re-announcing old posts by diffing scraped URLs against stored posts before upsert. This gates emails on genuinely new posts and fixes false positives (recoupable/chat#1855).

  • Bug Fixes
    • Added filterNewPostUrls and used it in the Instagram handler pre-upsert to detect genuinely new URLs.
    • Only send alert emails when the diff is non-empty; persistence and follow-up runs unchanged, with tests covering the helper and the no-email path.

Summary by CodeRabbit

  • New Features
    • Added smarter filtering so only newly discovered Instagram posts trigger follow-up notifications.
  • Bug Fixes
    • Prevented notification emails from being sent when scraped posts were already known.
    • Improved handling for empty scrape results by avoiding unnecessary notification attempts.