feat(ingest): accept Hookdeck-forwarded deliveries, and reject stale inbound Linear events

Merged
SA
3mo ago
+7218-22

Groundwork for routing webhook ingest through Hookdeck. Nothing here changes behaviour until a Hookdeck source actually exists — every delivery still arrives directly from the provider and verifies exactly as it does today.

Why

Hookdeck was chosen over building an ack-then-process gateway ourselves: at current volume it is roughly cost-neutral against the Vercel compute it displaces, and it buys burst absorption, retries, replay and a DLQ without a multi-week build. The full assessment is here.

Two things have to be true before any traffic moves, and both are in this PR.

1. Linear's ±60s replay window would reject every retry

parseVerifiedPayload hands Linear's webhookTimestamp to the SDK, which enforces Linear's ±60 second replay window. Hookdeck preserves the original provider signature and timestamp when it retries — so a delivery retried five minutes later arrives with a timestamp well outside the window and 401s permanently, on every attempt. The failure is silent, total, and lands precisely on the deliveries a gateway exists to save.

verifyHookdeckDelivery checks x-hookdeck-signature instead — HMAC-SHA256 over the raw body with no timestamp component, so it survives any delay. x-hookdeck-verified is checked alongside it and is not redundant: the signature proves Hookdeck sent the request, the flag proves Hookdeck authenticated the provider first. Without it, a source whose verification was never configured would forward whatever anyone posted to its URL, correctly signed and indistinguishable from real traffic.

GitHub needs no equivalent change — its signature covers no timestamp, so it already tolerates arbitrary retry latency. That makes it the right provider to pilot first.

2. The mirror applies stale inbound events

A write we push to Linear returns as a webhook and the mirror applies it unconditionally. This is not a reported failure and there is no evidence of it happening — the window is a second or two, and it is largely self-correcting because the second edit pushes too and its own echo repairs the damage.

But both of those protections are properties of low, ordered latency, which is exactly what a gateway removes. Queue depth widens the window to minutes on a retry, and at-least-once delivery makes reordering routine. The guard has to exist before deliveries can be delayed and reordered, not after.

Recording Linear's updatedAt on every outbound write and refusing anything not strictly newer covers the echo and the reordering with one comparison.

Verified

Provisioned a real Hookdeck source and pushed signed traffic through it rather than trusting the documentation:

  • x-github-delivery forwards byte-identical — the whole dedup story rests on this
  • a bad signature is rejected 401 at Hookdeck's edge
  • the forwarded delivery carries X-Hookdeck-Verified: true
  • recomputing HMAC-SHA256 over the raw body reproduced Hookdeck's X-Hookdeck-Signature exactly, and the real delivery passes end-to-end through the verifier

Also confirmed the rendered SQL for the conflict guard ("tasks"."external_updated_at" < excluded.external_updated_at, timestamps binding as ISO-8601 UTC) rather than assuming it.

Not covered

  • The migration has not been executed against a database — SQL rendered and inspected, not run
  • No Hookdeck source carries production traffic; the code path is inert until one does
  • HOOKDECK_SIGNING_SECRET is set from a trial project and must be rotated when a production project exists
  • Bounded retry for partial fan-out failure is still open — it should read Hookdeck's X-Hookdeck-Attempt-Count rather than our own retryCount

https://claude.ai/code/session_014D6wif8HDRCXEFf5Ku4w7V

Summary by cubic

Accepts Hookdeck-forwarded Linear webhooks and rejects stale or echoed Linear events. Old: we revalidated Linear’s ±60s timestamped signature and applied every event; New: we accept x-hookdeck-signature plus x-hookdeck-verified (retry-safe), persist external_updated_at, ignore events not strictly newer, and gate outbound updates so the watermark only moves forward. Behavior is unchanged until traffic routes via Hookdeck.

  • Review/Rollout
    • Dual path stays live: if x-hookdeck-signature is absent, we verify Linear’s signature as before. Rollback is URL repoint only.
    • Set HOOKDECK_SIGNING_SECRET in each environment; workflows pass it through. Signed but unverified/mismatched Hookdeck deliveries 401.
    • Migration: run 0091_tasks_external_updated_at. Existing rows start null and adopt a watermark on their next event.
    • Linear sync records provider updatedAt on writes and webhooks; outbound updates only advance the watermark (null-or-strictly-older), and inbound upserts gate on recency via SQL setWhere. GitHub webhook behavior is unchanged.

Summary by CodeRabbit

  • New Features

    • Added secure support for webhook deliveries routed through Hookdeck.
    • Linear task synchronization now tracks external update times.
    • Webhook processing filters stale, duplicate, and echoed updates to keep tasks current.

  • Bug Fixes

    • Improved Linear synchronization reliability for delayed or repeated webhook deliveries.
    • Preserved direct Linear webhook verification while supporting verified Hookdeck requests.

  • Security

    • Added HMAC-based validation and required verification checks for Hookdeck webhook traffic.
    • Enabled the signing secret for preview and production deployments.