feat(inbox): inbound email pipeline, generic provider, worker and message API (G-P4-2, part 2)

Merged
EV
10d ago
+5482-16

Plan item G-P4-2 (part 2 of 2) — Gauzy ↔ Ever Works integration plan, 04-inbox-and-email.md §2, §3, §5.2–5.5 / 06-gauzy-implementation-plan.md Phase 4. Also delivers the Inbox reply / forward half of G-P4-5 (they need inbox_email).

Owned by 7dae55c0/gw-gauzy-email — other lanes please don't push to this branch.
Stacked on #10803 (G-P4-2 part 1: provider registry + accounts); its commits show here until it merges. This PR's own work: the inbound/, emails/, mailboxes/, providers/generic/, testing/ folders and the five new migrations under packages/plugins/inbox/src/lib/email/, the contract additions, the env docs and EMAIL.md.

Everything is inside InboxEmailPlugin, listed only when GAUZY_INBOX_ENABLED=true and GAUZY_INBOX_EMAIL_ENABLED=true. Off (the default) = not loaded: no route, no table, no worker.

What

  • Tables (Postgres / MySQL / SQLite; idempotent, advisory lock on Postgres, down drops): inbox_mailbox (address unique across the deployment — a security control, mail is routed by address alone), inbox_mailbox_member, inbox_email_thread, inbox_email (unique on (mailboxId, messageId)), inbox_email_attachment. Secret-shaped columns are @ExportRedacted. Only varchar/text carry an explicit type (better-sqlite3).
  • Webhook POST /api/inbox/inbound/:providerId/:accountId (@Public(), per-client-address rate limit): the platform route /generic/platform (secret GAUZY_INBOX_PLATFORM_SECRET, ≥ 24 chars; unset = 404) or a provider account by id; provider.inbound.verify (401) then parse; per message only envelope recipients (else To/Cc) that are active mailboxes of this account (platform route: mailboxes without an account; same tenant, same org for an org account; custom domains only once verified; sender allowlist) — the mailbox row decides the tenant, never the payload; raw .eml (or normalized .json) to FileStorage once per tenant; one RECEIVED row per mailbox; 202 {accepted, duplicates, dropped}; > GAUZY_INBOX_MAX_MESSAGE_BYTES = 413.
  • Built-in generic signed webhook provider (what the Cloudflare Email Worker, G-P4-3, will post): X-Gauzy-Inbox-Signature: t=<unix>,v1=<hex> verified with @gauzy/core's verifyWebhookSignature (5 min), single use through InboundReplayGuard (Redis SET NX PX when REDIS_ENABLED, in-process otherwise); body = raw MIME (base64 + envelope + optional authenticationResults / spamScore) or a structured message; up to 20 per delivery.
  • Worker (in the API process: background task after the webhook + a 2-minute sweep for lost hand-overs / dead workers; conditional claim so replicas never double-process; 3 attempts then FAILED with the error; GAUZY_INBOX_EMAIL_WORKER=off opts a process out): server-side sanitize-html policy (no scripts/handlers/forms/frames/svg/style sheets; links noopener noreferrer nofollow; remote images blocked into data-remote-src; plain presentation styles only), attachments (SHA-256; HTML/SVG/XML/JS never offered as such), threading (In-Reply-To/References, else normalized subject + shared participant within 30 days), spam verdict (auth failures + provider score vs GAUZY_INBOX_SPAM_SUSPECT_SCORE / _QUARANTINE_SCORE) — quarantined mail gets no inbox item until a manager releases it, EMAIL inbox item through INBOX_ITEM_WRITER for the mailbox's recipients (employee owner's user; shared mailbox members; fallbacks: team members / org users whose role grants INBOX_MAILBOX_MANAGE), linked via mailboxId/emailId.
  • Message API /api/inbox/emails (INBOX_VIEW + mailbox access): GET /:id (sanitized HTML + attachment metadata), /:id/thread, /:id/raw (managers), /:id/attachments/:aid (always a download, nosniff, sandbox CSP), POST /:id/reply and /:id/forward (from the mailbox address, In-Reply-To/References, kept as OUT in the thread; sent through EmailSendService.sendMessage with the mailbox's provider account first → org/tenant default outbound → Custom SMTP / MAIL_*), GET /quarantine?mailboxId= + POST /:id/release (managers).
  • Deps: mailparser@^3.9.1 and sanitize-html@^2.18.0 (+ @types/sanitize-html@^2.16.2) — the exact ranges already in yarn.lock (mailparser comes through email-templates → preview-email today), so --frozen-lockfile needs no lockfile change. A local mailparser.d.ts types simpleParser.
  • Env docs (.env.sample, .env.docker, .env.compose): GAUZY_INBOX_DOMAIN, GAUZY_INBOX_PLATFORM_SECRET, limits, spam scores, worker switch. No Ever hostname literals added (egress static scan clean).

Not in this PR: mailbox management (allocation on GAUZY_INBOX_DOMAIN, custom-domain verification, forwarding, auto-provisioning, members API, docs namespace check) = G-P4-1, next; "Save to Documents" for attachments; the Settings → Inbox → Providers form (needs inbox-ui, sibling lane); BullMQ dispatch (the API has no scheduler root, so the docs-plugin "inline + sweep" model is used).

Does not touch InboxPlugin / InboxModule / InboxService (sibling lane gw-gauzy-inbox, #10796): the item's mailboxId/emailId are written on the item row right after upsertExternal, so this works with and without #10796's writer change.

Tests

  • Unit: addresses/subjects/allowlist, MIME parsing (multipart + inline + attachment, derived Message-ID, verdicts, JSON raw round trip), sanitizer, generic provider (signature, other secret, stale, tampered body, replay, no secret; raw + structured bodies).
  • inbox-inbound.db.spec.ts on better-sqlite3 with the real InboxService: webhook idempotency + +tag, account/mailbox rules, unverified domain + allowlist, 401/404/413, worker (sanitize, attachments, EMAIL item + recipients, employee mailbox, threading by header and by subject, quarantine + release, retries → FAILED, sweep), message API (read access incl. READ_ALL and other tenant, raw for managers only, reply with threading headers + scope, forward with attachments).
  • Not run locally (owner rule for this run: review + merge, full suites once on develop at the end). tsc --noEmit of plugin-inbox (sources + specs, strict, workspace path map) clean; prettier --check, cspell and tools/egress-audit/static-hostnames.mjs clean.

🤖 Generated with Claude Code

Summary by cubic

Adds the inbound email pipeline to the Inbox email plugin: provider webhooks receive mail, a worker sanitizes and threads it into inbox items, and a message API handles reading, replying, and forwarding.

The plugin stays off by default and loads only when both GAUZY_INBOX_ENABLED=true and GAUZY_INBOX_EMAIL_ENABLED=true are set; otherwise no routes, tables, or worker are registered. Since mailbox addresses are unique across the deployment, mail routing never depends on the payload — the mailbox row alone decides the tenant.

New Features

  • Six new tables with idempotent migrations across Postgres, MySQL, and SQLite: provider account, mailbox, mailbox member, email thread, email, and attachment.
  • POST /api/inbox/inbound/:providerId/:accountId webhook with a built-in generic signed provider (HMAC over the raw body) for the Cloudflare Email Worker. Verified signatures are single-use in Redis but released when storing fails so a provider's retry of the same signed request succeeds; the sender allowlist admits only DMARC-authenticated senders, an allowlisted sender without a DMARC pass is held in quarantine instead of dropped, and the rate limit keys on the client address Express resolves. An oversized batch is refused whole.
  • A background worker that sanitizes HTML, stores attachments (never offering active content), threads by In-Reply-To/References (fallback: normalized subject plus shared participant within 30 days), and quarantines spam until a manager releases it. Inbox item links pass through upsertExternal.
  • /api/inbox/emails API: sanitized read, thread, raw only for managers, attachment download, quarantine list and release, and reply/forward sent from the mailbox address through the provider accounts. A failed send stays retrievable as FAILED with its attachments, a forward carries every stored attachment, a sent message keeps the transport's own id when the provider reports one, and only users still active in the mailbox's organization count as recipients, readers, or managers.
  • Local installs keep email files in a private directory (GAUZY_INBOX_LOCAL_STORAGE_PATH, else ./storage/inbox), never under the public asset root. Raw messages are stored under a stable key (message id plus content digest), so simultaneous deliveries sharing the duplicate race leave one file behind. All email files are written and read with the deployment's storage settings in a fresh context, never the caller's tenant overrides; stream-returning storages are read into a buffer, and a failed upload throws instead of leaving a row pointing at a missing file.
  • Worker writes are guarded by the claimed attempt so a stalled worker cannot overwrite a newer retry, attachment keys are stable across retries, and thread assignment is serialized per mailbox. Attachment files upload before the claimed state and attachment rows are written in one transaction opened by the claimed update, so a taken-over worker neither deletes nor duplicates rows. Stored messages keep only allowlisted headers regardless of the delivery form, a sent reply/forward recounts its thread and moves its last activity, and provider-account edits that set a default take the scope lock before the row lock.
  • Thread counts now match what the thread view shows — a message is counted only once it has its final status, excludes quarantined and failed inbound work and deleted messages, and is recounted on release, terminal failure, and after a reply or forward.

Migration

  • New env vars: GAUZY_INBOX_EMAIL_ENABLED, GAUZY_INBOX_DOMAIN, GAUZY_INBOX_PLATFORM_SECRET, size and spam limits, GAUZY_INBOX_INBOUND_REQUESTS_PER_MINUTE, GAUZY_INBOX_EMAIL_WORKER, and GAUZY_INBOX_LOCAL_STORAGE_PATH.
  • New dependencies mailparser and sanitize-html are already in yarn.lock, so no lockfile change is required.