feat(backfill): Phase 2 rooms→sessions/chats/chat_messages script

Merged
SW
5mo ago
+602-1

Summary

Phase 2 of the chat→/api/chat/workflow cutover (recoupable/chat#1747): backfill legacy rooms + memories into the new sessions / chats / chat_messages model.

Built in api (not chat) because api owns this data model — so it reuses existing typed helpers and needs no type regeneration. Supersedes the chat-side attempt (recoupable/chat#1750), which had to hand-write helpers and regenerate chat's types because those tables aren't in chat's generated types.

What it does

  • scripts/backfillRoomsToSessions.ts + scripts/backfill/migrateRoom.ts — orchestration only, no raw supabase (per the repo's lib/supabase rule). For each room: upsert session → chat → messages.
  • Reuses insertSession / insertChat / upsertChatMessage, guarded by selectSessions / selectChats so re-runs are idempotent (deterministic uuidv5 session ids; room.id preserved as chat.id so /chat/[roomId] URLs keep working; upsertChatMessage({ update: false }) is write-once).
  • lib/supabase/rooms/selectAllRooms.ts — paginated all-rooms read (existing selectRooms caps at the PostgREST 1k limit; there are ~23.6k rooms). Reuses selectMemories per room.
  • Adds uuid dep + backfill:rooms-to-sessions script.

Run

SUPABASE_URL=… SUPABASE_KEY=… pnpm backfill:rooms-to-sessions

Idempotent — safe to re-run after a partial failure. Exits non-zero if any room fails.

Test plan

  • pnpm test lib/supabase/rooms/__tests__/selectAllRooms.test.ts — pagination (single page / multi-page / error) green
  • tsc --noEmit clean for the backfill files; import chain resolves under tsx
  • Dry-run against a non-prod project before the real run; reconcile counts (rooms→sessions, memories→chat_messages)

Notes

  • The transform assumes memories.content is { role, parts, content } (written by filterMessageContentForMemories). Pre-parts legacy rows would fail the parts NOT NULL upsert and be reported (non-zero exit), not silently dropped — worth a count of null-parts memories before the prod run.

🤖 Generated with Claude Code

Summary by cubic

Adds an idempotent backfill to migrate legacy rooms and memories to the new sessions/chats/chat_messages model, with concurrent room processing for faster runs. Supports dry-run, paginated reads, targeted runs (--limit, --room), and safe re-runs; keeps existing /chat/[roomId] URLs working.

  • New Features

    • Backfill orchestrator (scripts/backfillRoomsToSessions.ts, scripts/backfill/migrateRoom.ts) using typed helpers (no raw Supabase). Adds uuid, @types/uuid, and a backfill:rooms-to-sessions script.
    • Dry run (DRY_RUN=1 or --dry-run): reads and checks only; prints a full preview. Targeted runs: --limit=N, --room=<id>.
    • Pagination: generic scripts/backfill/paginate.ts + scripts/backfill/selectAllRooms.ts; selectRooms/selectMemories gain an optional range with stable id secondary sort to avoid skips/dupes (existing callers unchanged).
    • Idempotent re-runs: deterministic session IDs via uuidv5; preserve room.id as chat.id; write-once message upserts.
    • Performance: concurrent room pool (--concurrency=N, default 20) with Promise.allSettled isolation.
    • upsertChatMessages delegates to the single-row upsertChatMessage helper (no direct query); duplicates count as success; logs per-row failures and throws if any fail. Tests cover paginate, selectAllRooms, migrateRoom (dry/real/idempotent), and upsertChatMessages.

  • Migration

    • Run: SUPABASE_URL=… SUPABASE_KEY=… pnpm backfill:rooms-to-sessions (DRY_RUN=1/--dry-run, --limit=N, --room=<id>, --concurrency=N)
    • Safe to re-run; exits non‑zero if any room fails.

Summary by CodeRabbit

  • New Features

    • Added pagination support for memory and room queries via optional range parameters.
    • Added batch upsert functionality for chat messages with fallback handling.

  • Improvements

    • Enhanced data retrieval operations with deterministic ordering for consistent pagination results.

Review Change Stack