feat(apify): optional `posts` depth on social scrape endpoints

Merged
SW
4mo ago
+300-10

Summary

Implements recoupable/chat#1836. Contract: recoupable/docs#258 — merge order: docs → api (this PR).

Adds an optional posts parameter (integer, 1–100) to both scrape endpoints and threads it into the platform Apify actor inputs:

LayerChange
POST /api/socials/{id}/scrape?posts=N query param, validated in validatePostSocialScrapeRequest (400 on 0/101/non-integer, parsed before auth)
POST /api/artist/socials/scrapeoptional posts body field on artistSocialsScrapeBodySchema
scrapeProfileUrl / scrapeProfileUrlBatchnew optional posts arg forwarded to every platform ScrapeRunner
Twitter (apidojo/twitter-scraper-lite)maxItems: posts ?? 1 — last N tweets with engagement metrics incl. viewCount (impressions)
YouTube (streamers/youtube-scraper)maxResults: posts ?? 1, maxResultsShorts: posts ?? 0 — a requested depth includes Shorts, which the legacy snapshot excluded
Other platformsaccept and ignore the param (Instagram profile scrapes already return latestPosts)

Backwards compatible / cost-neutral by default: omitting posts produces the exact same actor inputs as before (unit-tested), so existing profile-snapshot callers and their Apify billing are unchanged.

Tests (TDD, red → green)

All new behavior was written test-first and confirmed failing (20 failures across 8 files) before implementation:

  • startTwitterProfileScraping: default input unchanged; posts=20 → maxItems: 20
  • startYoutubeProfileScraping: default input unchanged (Shorts excluded); posts=20 → maxResults/maxResultsShorts: 20
  • scrapeProfileUrl + scrapeProfileUrlBatch: posts forwarded (and undefined when omitted)
  • validatePostSocialScrapeRequest: posts=20 parses; 0/101/abc/1.5 → 400 before auth
  • validateArtistSocialsScrapeBody + postArtistSocialsScrapeHandler: new suites (missing id → 400, posts pass-through, no-socials → [])

Full suite: 3745/3745 pass. Affected domains (lib/apify, lib/socials, lib/artist): 234/234. tsc --noEmit introduces no new errors (200 with this diff vs 203 pre-existing on base). ESLint clean on touched dirs.

Preview verification against the live deployment to follow as a PR comment.

🤖 Generated with Claude Code


Summary by cubic

Add optional posts depth to social scrape endpoints so callers can request up to N posts while keeping legacy single-item behavior by default (cost-neutral).

  • New Features

    • POST /api/socials/{id}/scrape: accepts ?posts=1..100; invalid values return 400 before auth.
    • POST /api/artist/socials/scrape: accepts optional posts body field (1–100).
    • Threads posts through scrapeProfileUrl and scrapeProfileUrlBatch into platform scrapers.
    • Twitter (apidojo/twitter-scraper-lite): maxItems = posts ?? 1.
    • YouTube (streamers/youtube-scraper): maxResults = posts ?? 1, maxResultsShorts = posts ?? 0; when provided, Shorts are included.
    • Other platforms accept and ignore posts (no behavior change).
  • Migration

    • No changes required for existing callers.
    • To request depth: use ?posts=N on /api/socials/{id}/scrape or { posts: N } on /api/artist/socials/scrape (1–100).