fix(db,ci): make migration 0119 online-safe and bound production migrations

Merged
KI
2mo ago
+30-9

Summary

Unblocks production deploys after the 17 September API outage. Deploy Production is still disabled; this PR is what has to merge before it is turned back on.

  • Migration 0119 no longer rewrites ingest.webhook_events. The enum to text change on provider was a full rewrite of the largest table in production, run while the same transaction held ACCESS EXCLUSIVE locks on auth.users, auth.organizations and integration_connections. 0119 is regenerated with drizzle-kit generate and is identical to the original minus that one statement. It rolled back during the incident and never applied in production.
  • provider stays the integration_provider enum. Every writer passes linear, github, notion or slack, all already in the enum. recordWebhookDelivery goes back to the enum type, so a non-enum provider is a compile error.
  • Production migrate job is bounded. A concurrency group (three migrators ran at once during the incident) and PGOPTIONS: -c lock_timeout=5s -c statement_timeout=60s. drizzle applies every pending migration in one transaction, so splitting files does not isolate locks; timeouts are what make a blocked migration fail and roll back instead of stalling traffic.
  • Preview migrate step gets the same timeouts, so the preview database check runs what production will run.

Testing

Rehearsed with the exact CI command on a Neon branch of production (122 migrations applied, 540 GB webhook_events, 2 CU):

ScenarioResult
No timeouts, one held read lock on auth.usersMigrator still waiting on DROP TABLE ... CASCADE after 12s, killed
With timeouts, same blockerExit 1 after 7.0s, canceling statement due to lock timeout, clean rollback, still 122 applied
Clean runExit 0 in 2.3s, new tables present, old table and FK gone, provider still the enum
Re-runNo-op, exit 0

PGOPTIONS confirmed to reach the session through Neon's direct endpoint with the @neondatabase/serverless driver. db and api typecheck, ingest tests, lint and drizzle-kit check pass. The Deploy Database (Neon) check on this PR applies 0119 to a fresh branch of production with the same timeouts.

Notes

  • The journal timestamp for 0119 is kept, so a dev database that already applied the old 0119 skips this one. Its provider column stays text until the branch is reset; inserts still work.
  • #7589 carries its own 0119_slack_agent_launches.sql and needs a rebase and regenerate.
  • The concurrency group gets its first real exercise on the first merge after the workflow is re-enabled.

Summary by cubic

Fixes the 17 September production outage by regenerating migration 0119 without the webhook_events table rewrite and ordering its locks to match the application. Unblocks production deploys; Deploy Production stays disabled until this merges. Production migration runs are now bounded and retried.

Bug Fixes

  • 0119 no longer changes provider to text; the column stays the integration_provider enum, and recordWebhookDelivery types it as the enum so invalid providers fail compilation.
  • The automation_events foreign key is dropped before plugin_connections, child table first like inserts, avoiding the deadlock seen under traffic.
  • The journal timestamp for 0119 is kept, so dev databases that already applied the old 0119 skip this one.

Reliability

  • Production migrations run under a concurrency group with lock_timeout=5s and statement_timeout=60s via PGOPTIONS; preview gets the same timeouts.
  • Because drizzle applies pending migrations in one transaction, those timeouts make a blocked migration roll back instead of holding traffic.
  • The production migrate step makes up to eight attempts, retrying only on deadlock or lock timeout.

Summary by CodeRabbit

  • Bug Fixes

    • Webhook delivery records now accept only supported integration providers, preventing invalid values.
    • Webhook event data now consistently enforces the supported provider set at the database level.

  • Reliability

    • Database deployments now apply lock and execution time limits to reduce stalled migrations.
    • Production migrations run one at a time and retry select transient failures automatically.
    • Migration ordering has been adjusted to reduce deployment conflicts.