feat(apify): persist scrape results for the remaining 5 platforms
Merged
SW
4mo ago+506+506+012345678901234567890123456789-16-16-01234567890123456789
Part of recoupable/chat#1833 — completes the persistence work from api#738 (registry + LinkedIn). Base: main.
What
TikTok, X/Twitter, YouTube, Threads, and Facebook scrapes now persist to socials instead of being poll-only. Per platform, mirroring lib/apify/linkedin/:
- Webhook attachment — each
start…module passes{ webhooks: getApifyWebhooks() }(root cause of "nothing persists": only IG/LinkedIn runs ever called/api/apify). - Result handler —
handle<Platform>ProfileScraperResultsupsertssocials(keyedprofile_url), mapping only fields observed in a real actor run of each platform (run ids in the code comments: TikTokG4YRI0e…, XALVMZYX…, YouTubeH0ZrIAs…, Threads9iiG1sD…, FacebookICZxdJB…). - Registry entry — actor ids resolved live from the Apify API:
GdWCkxBtKWOsKjdch(tiktok),nfp1fpt5gUlBwPcor(x),h7sDV53CddomktSi5(youtube),kJdK90pa2hhYYrCK5(threads),4Hv5RhChiaDk6iwad(facebook).
Round-trip hazards handled explicitly
upsertSocials keys on exact profile_url, so each handler must emit a URL that matches the stored row:
- X/Twitter: the actor echoes display casing (
x.com/TheASF) while stored rows are lowercase — the handler lowercases (X handles are case-insensitive). Without this, every scrape would create a duplicate row. - YouTube: the actor's
channelUrlis the/channel/UC…form, which never matches the stored@handlerow — the handler keys oninputChannelUrl(the exact URL this service passed to the actor).
Tests (TDD)
- 5 handler test files, each asserting the exact upsert from its real trimmed fixture + empty-dataset no-op.
- Registry test extended to all 5 new actor ids; webhook-handler test unchanged in behavior (new handler mocks only).
- Full
lib/apifysuite green (65 tests / 20 files); tsc + eslint clean.
Preview verification (live scrape → webhook → DB diff on real rows) to follow as a comment.
🤖 Generated with Claude Code
Summary by cubic
Persist TikTok, X/Twitter, YouTube, Threads, and Facebook scrape results to socials via Apify webhooks, completing the remaining platform persistence. This satisfies recoupable/chat#1833 and removes the poll-only behavior.
-
New Features
- Added
handle<Platform>ProfileScraperResultsfor all 5 platforms to upsert intosocialskeyed byprofile_url. - Attached
getApifyWebhooks()to allstart…modules so runs hit/api/apify. - Registered each platform’s actor ID in
getApifyResultHandler. - Tests added for each handler and the registry mapping.
- Added
-
Bug Fixes
- X/Twitter: lowercased
profile_urlto avoid duplicate rows caused by display casing. - YouTube: upsert keyed on
inputChannelUrl(matches stored@handle), not/channel/UC…URLs.
- X/Twitter: lowercased