feat(api): migrate apify webhook + instagram scrape chain

Merged
AR
6mo ago
+1538-57

Ports chat's Apify webhook receiver and Instagram profile/comments scrape handlers into mono/api. Follow-up scrapes now run as in-process function calls — no HTTP hops between chat and api for the Apify chain.

Test plan

  • Unit tests cover webhook validator (accept / reject / bad JSON) and each ported handler (profile success, comments success, dispatcher branching, post/social upsert helpers)
  • Preview smoke: POST https://<preview>/api/apify with a stub payload returns 200
  • lib/supabase/post_comments/ and lib/supabase/social_posts/ helpers typecheck against SupabaseClient<Database>
  • Real end-to-end Apify run against preview — to be verified post-merge to test (Apify actors currently call the chat webhook; flip is part of a follow-up)

Summary by cubic

Migrates the Apify webhook and Instagram scraping pipeline into mono/api, replacing cross-service HTTP with in-process calls and adding stronger SSRF protections and timeouts. Adds POST /api/apify, standardizes webhook URLs, tightens validation, and always replies 200 to avoid Apify retries.

  • New Features

    • POST /api/apify receiver (apifyWebhookHandler) with schema validation and inline dispatch; always 200 with { status: "error" } on invalid/unhandled payloads or handler errors.
    • Profile results: upsert posts/socials, link social_posts, mirror profile pics to Arweave (SSRF-safe, 10MB cap, 10s fetch timeout), email subscribed accounts, and queue comments scraping with cheap refresh for posts that already have comments.
    • Comments results: upsert post_comments, auto-create missing posts and commenter socials, and queue fan profile scrapes for distinct usernames.
    • Starters: startInstagramProfileScraping (1+ handles, deduped) and startInstagramCommentsScraping, both registering webhooks via getApifyWebhooks (prod alias, preview self, none locally).

  • Refactors

    • Webhook base URL via getBaseUrl + getApifyWebhooks; validator requires non-empty actorId/defaultDatasetId and always returns a 200 error body; dataset fetches use the Apify SDK and bubble errors.
    • Security: isSafeHttpUrl blocks localhost/private/CGNAT/multicast and IPv4-mapped IPv6; uploadLinkToArweave enforces 10s/10MB.
    • Supabase: idempotent upserts; insertSocials upserts on profile_url and drops nullish; new posts, post_comments, and social_posts helpers; selectPostUrlsWithComments composes getPosts + selectPostComments.
    • Comments persistence inlined into handleInstagramCommentsScraper.
    • Build: resolved TS types in comments handler and insertSocials to unblock Vercel build.

Summary by CodeRabbit

Release Notes

  • New Features
    • Added Instagram profile and comment scraping capabilities with automatic data collection
    • Implemented automatic profile picture archival to decentralized storage
    • Added email notifications when profile data updates are processed
    • Integrated webhook support for external data pipeline automation