feat(apify): persist scrape results for the remaining 5 platforms

Merged
SW
4mo ago
+506-16

Part of recoupable/chat#1833 — completes the persistence work from api#738 (registry + LinkedIn). Base: main.

What

TikTok, X/Twitter, YouTube, Threads, and Facebook scrapes now persist to socials instead of being poll-only. Per platform, mirroring lib/apify/linkedin/:

  1. Webhook attachment — each start… module passes { webhooks: getApifyWebhooks() } (root cause of "nothing persists": only IG/LinkedIn runs ever called /api/apify).
  2. Result handler — handle<Platform>ProfileScraperResults upserts socials (keyed profile_url), mapping only fields observed in a real actor run of each platform (run ids in the code comments: TikTok G4YRI0e…, X ALVMZYX…, YouTube H0ZrIAs…, Threads 9iiG1sD…, Facebook ICZxdJB…).
  3. Registry entry — actor ids resolved live from the Apify API: GdWCkxBtKWOsKjdch (tiktok), nfp1fpt5gUlBwPcor (x), h7sDV53CddomktSi5 (youtube), kJdK90pa2hhYYrCK5 (threads), 4Hv5RhChiaDk6iwad (facebook).

Round-trip hazards handled explicitly

upsertSocials keys on exact profile_url, so each handler must emit a URL that matches the stored row:

  • X/Twitter: the actor echoes display casing (x.com/TheASF) while stored rows are lowercase — the handler lowercases (X handles are case-insensitive). Without this, every scrape would create a duplicate row.
  • YouTube: the actor's channelUrl is the /channel/UC… form, which never matches the stored @handle row — the handler keys on inputChannelUrl (the exact URL this service passed to the actor).

Tests (TDD)

  • 5 handler test files, each asserting the exact upsert from its real trimmed fixture + empty-dataset no-op.
  • Registry test extended to all 5 new actor ids; webhook-handler test unchanged in behavior (new handler mocks only).
  • Full lib/apify suite green (65 tests / 20 files); tsc + eslint clean.

Preview verification (live scrape → webhook → DB diff on real rows) to follow as a comment.

🤖 Generated with Claude Code


Summary by cubic

Persist TikTok, X/Twitter, YouTube, Threads, and Facebook scrape results to socials via Apify webhooks, completing the remaining platform persistence. This satisfies recoupable/chat#1833 and removes the poll-only behavior.

  • New Features

    • Added handle<Platform>ProfileScraperResults for all 5 platforms to upsert into socials keyed by profile_url.
    • Attached getApifyWebhooks() to all start… modules so runs hit /api/apify.
    • Registered each platform’s actor ID in getApifyResultHandler.
    • Tests added for each handler and the registry mapping.
  • Bug Fixes

    • X/Twitter: lowercased profile_url to avoid duplicate rows caused by display casing.
    • YouTube: upsert keyed on inputChannelUrl (matches stored @handle), not /channel/UC… URLs.