fix(ci): a failed production migration can no longer pass as a success
Summary
Follow-up to #7671. Deploy Production is disabled again until this merges.
On the first production run after #7671 (run 35466469044), the migration hit its 5 second lock timeout on automation_events, rolled back cleanly and drizzle-kit exited 1, which is the designed safe failure. The step still reported success, so the app deploys started against an unmigrated database. The run was cancelled before any app job reached its build step, so no app code shipped. Production stayed at 122 migrations applied and healthy throughout (peak 38 lock waiters for 3 seconds, connections flat at 77 to 79).
Cause: the retry loop added in #7671 pipes drizzle-kit migrate through tee to grep for a lost lock race. GitHub runs a step with no explicit shell: as bash -e {0}, without pipefail, so the pipeline's exit status was tee's. It was tested locally under bash -eo pipefail, which GitHub only uses when a step sets shell: bash.
Fix: the block sets pipefail itself.
Testing
Run with /bin/bash -e, the shell from the production log, against a Neon branch of production with 0119 pending:
| Case | Result |
|---|---|
| Old block, lock held 12s | exit 0 after 6.3s, nothing applied (reproduces the bug) |
| New block, lock held past all 8 attempts | exit 1 after 228s, nothing applied, 8 retry warnings |
| New block, bad credentials | exit 1 after 0.7s, no retries |
| New block, lock held 14s | one retry, exit 0, applied 123 |
| New block, nothing pending | exit 0 |
Summary by cubic
Fixes the production migration job so a failed migration can no longer pass as successful and start app deploys against an unmigrated database. The retry loop now sets pipefail, so the step exits with drizzle-kit's status instead of tee's when the lock times out. A new verification step fails the job unless the newest migration in the repo's journal is recorded in the database, and preview deploys run the same check. Deploy Production is disabled until this merges.
Summary by CodeRabbit
- Chores
- Improved deployment reliability by ensuring database migration failures are detected correctly.
- Failed migrations now properly trigger the configured retry behavior instead of being incorrectly reported as successful.
- Added post-migration checks to confirm that the production and preview databases are fully up to date before deployment continues.