Retain post captions, media references and publish dates

SI
5d ago
+346-0

Summary

Adds four additive columns to public.posts — caption, published_at, media (jsonb array of provider media references, NOT NULL default [], CHECK (jsonb_typeof(media) = 'array') added NOT VALID and validated in a second file 20261010213201_posts_media_retention_validate.sql so the existing-row scan does not hold the ALTER TABLE lock) and media_observed_at — so the existing Instagram profile scraper can retain the captions, post images/carousels and publish dates it already reports instead of discarding them. This is the storage half of the ticket's first Done-when item ("artist images and captions remain retrievable with source/version/date/usage status"). Entries are provider URLs only (a video entry's URL is its cover still): no bytes are retained, nothing is backfilled, NULL caption (not captured) stays distinct from '' (platform returned empty), and the existing posts contract (updated_at, engagement columns, recoupable/docs#316) is untouched. Includes a disposable-PostgreSQL transaction test, one added CI line in release-cases.yml and a tests/README.md section.

Refs recoupable/app#2132 (child of epic recoupable/app#2116).

Merge sequencing

docs → database → api → app. No docs change in this slice (no public HTTP contract changes). This database PR merges and both migrations are applied first. recoupable/api#1014 (branch claude/ctx-2132-social-media-retention) writes and reads the new columns and must not deploy before they are applied: its Instagram writer names the new columns for nearly every post and PostgREST rejects unknown columns, so artist post persistence would fail. The api currently on main does not use these columns and keeps working before and after. Writers must send uniform column sets per bulk upsert (PostgREST NULL-fills omitted keys, which media NOT NULL refuses); api#1014 groups rows by column set. No dependency on any OPEN PR; this migration only alters public.posts, which none of database#88–#95 touch. Based on current main (includes the #96/#98 CI edits).

Verification

Latest run, after the review fixes in c9b8483 (NOT VALID + separate validate file, PostgREST-shaped upsert tests, cover-still docs):

  • Ran: export PATH="/opt/homebrew/opt/postgresql@17/bin:$PATH" LC_ALL=C; python3 -m unittest discover -s tests -p 'test_post_media.py' → 10 passed, 0 failed, 0 skipped.
  • Ran: python3 -m unittest discover -s tests -p 'test_release_cases.py' → 1 passed; python3 -m unittest discover -s tests -p 'test_company_assessment.py' → 3 passed.
  • The suite now runs the statement PostgREST uses for a supabase-js bulk upsert: the column list is the union of row keys, and an omitted key arrives as NULL. It covers three cases. A re-scrape row without caption/media keeps the retained values. One upsert that mixes a carousel row with media and a legacy row without media is refused (null value in column "media"), and the retained legacy values are untouched. Upserts grouped by column set succeed, keep the legacy caption/media, and default a new row's media to []. This replaces the earlier hand-written single-row ON CONFLICT test, which did not match PostgREST's shape.
  • Red → green (review fix): against the previous single-file migration (inline CHECK), the new test_check_is_added_not_valid_and_validated_by_the_second_file failed. After the constraint is dropped, the old file does not add it back NOT VALID; this test also removes the constraint, so two dependent tests failed too, 3 failures in total. All 10 pass with the split migration. The test now restores the constraint in a cleanup step.
  • Red → green (original slice): test_post_media.py failed in setUpClass with FileNotFoundError before the migration existed.
  • Local note: on macOS the postmaster needs LC_ALL set ("postmaster became multithreaded during startup"); documented in tests/README.md.
  • Not verified: hosted deployment, application to any hosted/production database, a running PostgREST (its statement shape is reproduced in SQL), live providers (no scrape was run), PostgreSQL 15/16 locally (the CI matrix covers them; the local run used 17.7).

Out of scope / known gaps

  • No backfill: rows scraped before this migration keep caption NULL and media [] until the next scrape.
  • No retained bytes, storage path or receipts (that belongs to the private-original work in api#986–#991 / database#94–#95); no expired-URL refresh; no production-asset approval state; no era tables or RPCs; no RLS or grant changes.
  • Reported state: implemented + fixture-tested only. Not connected, deployed or live-verified.

🤖 Generated with Claude Code