feat(web): 90-day-ready email content cleanup: batched, safe on large tables, runs on cloud when enabled
Why
Cloud keeps every email's HTML, text, attachments and headers forever. That's 3.7 GB of the 7.1 GB production database, and it grows about 250 MB a month. We're setting a 90-day retention for email content on cloud. Delivery status and events are kept.
This PR changes nothing on its own. The cleanup only runs where EMAIL_CLEANUP_DAYS is set, and on cloud it'll be set only after customers are told. The one thing that applies on deploy is the autovacuum tuning on Email, which changes a table setting and touches no data.
What it does
- Rule: clear
html,text,attachmentsandheadersof emails sent more thanEMAIL_CLEANUP_DAYSago, meaning created and last updated before the cutoff. Sending updates the row, so an email that waited in a queue keeps its content for N days after it went out. SES status updates don't touchupdatedAt. - Never cleared: scheduled or queued emails, and emails scheduled after the cutoff.
- Nightly pace: walks forward by
createdAtfrom a saved position, so it never rereads emails it has already looked at. It clears at most 100,000 emails a night, in batches of 500. Each batch is its own transaction withlock_timeout 5sandstatement_timeout 60s, there's a 15-minute limit per run, and short pauses between batches. - Skipped emails: emails looked at but not clearable yet go into a Redis sorted set, scored by when they become eligible, and are rechecked by id once due. Production Redis is
noevictionwith nomaxmemory, so the set can't be evicted on its own. - Untouched fields: doesn't change
updatedAt, which the campaign page sorts by. - Vacuum: no
VACUUMfrom the app. A migration setsautovacuum_vacuum_scale_factorandautovacuum_analyze_scale_factorto 0.02 onEmail, inside a transaction withlock_timeout 10s. That makes autovacuum visit after about 2% of rows change, instead of 20%. - Dashboard: the email page explains removed content ("removed N days after sending; status and events are kept") instead of "No content to preview".
- Self-hosted: the same job replaces the old single unbatched
UPDATE. The self-hosting docs are updated.
Tested
- Real Postgres (
cleanup-email-bodies.integration.test.ts, runs in CI):- The right emails are cleared, and recent, scheduled, queued and late-sent ones are kept.
updatedAtis untouched.- A late-sent email is rechecked exactly when due (not at 70 days, cleared at 81 days on a simulated clock).
- Ties on
createdAtacross batches are all handled. - The cap holds exactly, in both the recheck and scan phases.
- Removing the
updatedAtcondition fails 3 tests.
- Scale: Postgres 16 with 400k production-shaped emails (~4 KB TOASTed bodies).
- One night of 50k took 23 s and wrote 41–110 MB of WAL.
- With autovacuum tuned, two nights of 100k left no dead tuples, and the table was 278 MB smaller.
- The keyset batch query is an
Index Scan BackwardoncreatedAtplus anIncremental Sort, about 30 pages per batch.
- Migrations: apply on a fresh database, and
prisma migrate diff(database → schema) is empty. - Suites: unit (376) and tRPC (23) tests pass, and the production build passes.
- Review: Astra reviewed three rounds, and every finding is addressed.
Turning it on (cloud, later)
- Customer notice, plus privacy policy and docs updates (drafts to be approved separately).
- Set
EMAIL_CLEANUP_DAYS=90on Railway and redeploy. - Watch the first nights:
[EmailCleanupJob]logs, dead tuples, WAL and free disk.
The backlog of about 1.2M emails clears in about 12 nights.
Ask Polylane for an impact assessment of this pull request:
Summary by cubic
Rebuilds the email content cleanup so it works on cloud at scale, and explains removed content in the email dashboard.
The old job was a single UPDATE over all old emails and only ran on self-hosted installs. The new job runs wherever EMAIL_CLEANUP_DAYS is set and works in small nightly batches, so nothing changes until that env var is set. The only change that applies on deploy is a migration that tunes autovacuum on Email — it alters a table setting and touches no data.
- Walks emails forward by
createdAtwith a saved position, clearing at most 100,000 per night in batches of 500, each batch its own transaction withlock_timeout5s,statement_timeout60s, and a 15-minute run limit. - Clears
html,text,attachments, andheadersonly for emails created and last updated before the cutoff, so scheduled, queued, and late-sent emails keep their content; delivery status and events are always kept. - Emails looked at but not ready yet are tracked in a Redis sorted set (scored by when they become eligible) and rechecked by id once due.
- Does not touch
updatedAt, which the campaign page sorts by. - Replaces the app-level
VACUUMwith autovacuum tuning onEmail(autovacuum_vacuum_scale_factorandautovacuum_analyze_scale_factorat 0.02). - The email details page explains that content was removed after N days instead of showing "No content to preview", and the self-hosting docs are updated.
Summary by CodeRabbit
- New Features
- Email content is cleared after the configured retention period, while delivery status and event history are retained.
- Scheduled or queued emails keep their content until sending. Older eligible emails may be cleaned up gradually, with a nightly limit.
- Email previews show a retention notice when content has been removed.
- Documentation
- Updated self-hosting guidance with details about retention, cleanup limits, and freed storage.