Retain Instagram captions and media as reference-only social evidence

SI
5d ago
+785-34

Summary

Stops discarding the captions, post images/carousels and publish dates the existing Instagram profile scraper already returns, and surfaces them, reference-only, through the saved social_context evidence module. New mapInstagramPostMedia turns one latestPosts item into {caption, published_at, media[]} (carousel children in order with kind/dimensions/alt, deduped by provider URL, capped at 20, missing fields stay null — never invented) and mapInstagramPostsToRows spreads it into the posts upsert, naming retention columns only when the run reported them. upsertPosts then writes each distinct column set in its own call: postgrest-js names the union of all rows' keys and PostgREST fills a key a row omits with NULL, so one bulk upsert of a run mixing posts with and without media would violate posts.media NOT NULL (losing the run's posts) or overwrite a retained caption with NULL. Grouped writes mean a degraded re-scrape never erases retained evidence. A separate selectPostMedia reader keeps selectPosts and the public GET /api/artists/{id}/posts contract (recoupable/docs#316) unchanged. collectContextSocials merges the rows per post as usage_status: reference_only, retained_bytes: null, provider_url_status: may_expire, replaces the hard-coded "post images, videos and captions are not returned" gap with a computed "N of M posts have no retained caption/media" gap, keeps the "not verified as official" gap, tolerates a failed media read (posts kept, gap added, error text not leaked) and records a trace.mediaRetention note that only provider URLs are kept. This advances the ticket's first Done-when item (images/captions retrievable with date and usage status) and keeps release artwork, artist brand and social imagery as separate topics. No scraping, no model call, no bytes, no new HTTP route, no spend.

Refs recoupable/app#2132 (child of epic recoupable/app#2116).

Merge sequencing

docs → database → api → app. No docs change (no public HTTP contract changes). Depends on recoupable/database#104 (20261010213200_posts_media_retention.sql + 20261010213201_posts_media_retention_validate.sql). Do not deploy this api change until both migrations are applied. The Instagram writer names published_at for nearly every post and caption/media/media_observed_at for nearly every post with a caption or image, and PostgREST rejects columns that do not exist, so on an unmigrated posts table upsertPosts throws and Instagram artist post persistence (posts, social_posts links, follow-up runs) fails. The reader side degrades more gently: selectPostMedia fails and social_context reports "Saved post captions/media could not be read". No dependency on any OPEN api PR: this slice adds no dispatcher action, RPC or lib/mcp change; the lib/context/README.md edit is three additive lines under "Scope and costs" to keep rebases against api#983/#985 trivial. Byte retention stays with api#986–#991 (retained_bytes: null here). No app PR for this slice (no UI change).

Plan adjustments while implementing (vs. the scoping plan): the media dependency of collectContextSocials is required rather than optional (silently skipping the read would be dishonest); published_at is retained whenever the actor reports a timestamp even when caption/media are absent, while media_observed_at is only stamped when caption or media were actually written; the row shape type lives in lib/posts/PostMediaEntry.ts; the migration's CHECK is named posts_media_is_array.

Verification

Latest run, after the review fixes in a720aeac (grouped upserts, access-denied test, cover-still docs):

  • Ran: pnpm exec vitest run lib/context lib/apify lib/supabase/posts → 104 files passed, 8 skipped; 492 tests passed, 0 failed, 8 skipped.
  • Ran: pnpm exec vitest run lib/context → 330 tests passed, 0 failed, 8 skipped.
  • Ran: pnpm exec tsc --noEmit → 199 errors, all in pre-existing unrelated test files; none in lib/supabase/posts, lib/apify, lib/context/providers, lib/posts or types/database.types.ts. The same total as origin/main (the original per-file diff against a detached origin/main checkout was empty).
  • Ran: pnpm exec eslint on the 6 changed source files and 6 changed test files → clean; pnpm exec prettier --check on all 14 changed files → clean.
  • Red → green (review fix): against the previous upsertPosts, the new tests in lib/supabase/posts/__tests__/upsertPosts.test.ts failed 2 of 4: a mixed batch made 1 upsert call instead of 2 (so PostgREST would NULL-fill), and an empty batch crashed. Both pass after grouping. A third test, added after the fix, feeds mapInstagramPostsToRows output for a mixed carousel and legacy run through upsertPosts and asserts that no call would be NULL-filled. recoupable/database#104 now runs the PostgREST statement shape on local PostgreSQL 17. One upsert that mixes media and no-media rows fails with the NOT NULL violation. Upserts grouped by column set keep retained values.
  • Red → green (original slice): before implementation mapInstagramPostMedia.test.ts and selectPostMedia.test.ts failed to load (module missing), and 7 tests failed. handleInstagramProfileScraperResults.test.ts had 2 failures: the upsert rows lacked caption, media and published_at. socialContext.test.ts had 4: the fixed gap was still present, the media reader was never called, caption and media were not merged, and there was no read-failure gap. socialCollection.test.ts had 1: the saved content lacked usage_status: reference_only.
  • Not verified: hosted deployment, live Apify runs (no scrape executed; fixtures are synthetic URLs/handles), production or hosted database, hosted Context enrichment saves, real PostgREST HTTP round-trip (the NULL-fill behaviour is taken from the installed @supabase/postgrest-js@2.86.0 source and reproduced in SQL, not against a running PostgREST).

Out of scope / known gaps

  • Provisional era summaries with overlapping dates, artist-vs-release scope and customer overrides (needs model synthesis = paid, and #2130).
  • Expired-URL repair without re-analysis (needs a refresh action keyed by content fingerprint), retained asset bytes/receipts (api#986–#991), explicit production-asset approval state.
  • Other platforms' handlers (TikTok, X, YouTube, LinkedIn) still discard their media fields; only the Instagram profile path retains them here.
  • A video media entry's provider_url is the cover still the scraper reports; the video stream URL is not retained. Reference-only status is recorded per post, not per media entry.
  • Grouped upserts are sequential, not one transaction: if a later group fails, earlier groups are already written. upsertPosts still throws, and a webhook replay re-runs the idempotent upserts.
  • Docs (no public contract change) and chat UI.
  • Reported state: implemented + fixture-tested only. Not connected, deployed or live-verified.

🤖 Generated with Claude Code

Summary by cubic

Retains the captions, publish dates, and media references the Instagram profile scraper already returns, and surfaces them reference-only in the saved social_context evidence module.

  • mapInstagramPostMedia converts a post into caption, publish date, and media entries in carousel order, deduped by provider URL and capped at 20; missing fields stay null, nothing is invented.
  • Retention columns are only written when the run reported them, so a degraded re-scrape never erases a caption or media an earlier run retained.
  • A separate selectPostMedia reader keeps the public GET /api/artists/{id}/posts contract unchanged.
  • collectContextSocials merges the rows per post as usage_status: reference_only, retained_bytes: null, provider_url_status: may_expire, and replaces the hard-coded "imagery not returned" gap with a computed gap counting posts without retained caption/media.
  • Requires recoupable/database#104 (20261010213200_posts_media_retention.sql); until it's applied, upserts still work but the media reader fails and social_context reports a gap instead of erroring.
  • No scraping, no model calls, no bytes stored, no new HTTP route.