fix(entitlement): race-safe grant and activation on a late order.placed

Merged
EV
14d ago
+3095-128

Summary

Closes the two gaps #11023 lists in the entitlement grant path:

  • A. grant() is now race-safe. A new migration adds UQ_entitlement_live_line_grant, one live (not revoked, not deleted) right per tenant, order line and kind. grant() answers the database's refusal by returning the winner's right as a replay. A line can still be granted again after its right is revoked: a derived number that the revoked right still holds takes the next revision (…-L-2).
  • B. A late order.placed follows where the order is now. The placement asks the order package for the order's state:
    • already cancelled or refunded in full: it grants nothing and logs why;
    • settled since: it puts into force, line by line, the rights of the lines still paid for in full, at their granted quantity. A line refunded in full or in part before the placement keeps its right PENDING.

Design (confirmed by the owner): B uses option 2 from the issue, with the existing order and order-line rows as the durable memo instead of a new memo table.

  • order.completed is appended in the same transaction that writes status = COMPLETED, so the row cannot disagree with the event.
  • order_line.refundedQuantity is the order package's own cache over its succeeded refunds.
  • The entitlement package does not read those tables itself. It reads them through a new optional port that the order package answers, following the same pattern as ENTITLEMENT_CATALOG_PORT.

Why

#11023, follow-up of #10914:

  1. The grant consumer is at-least-once and REORDERABLE. When order.completed / payment.captured is handled before a retried order.placed, the settlement finds nothing PENDING to activate. The rights the late placement then grants stayed PENDING, and no event was left to move them.
  2. grant() read before it inserted (findExistingGrant), and nothing in the database backed that read: IDX_entitlement_order_line is not unique. Two concurrent grants of one line could both insert, giving one purchase two licences. When the numbering series is unreachable, both grants derive the same number, so the second failed on UQ_entitlement_number with a raw driver error instead of a replay. The same derived-number collision also made a re-grant after revocation fail.

What changed (by package)

@gauzy/plugin-entitlement

  • database/migrations/1791000003580-AddEntitlementLiveLineGrantUniqueIndex.ts (new), registered in ENTITLEMENT_MIGRATIONS:

    • PostgreSQL / SQLite: partial unique index on ("tenantId", "orderLineId", "kind") WHERE "orderLineId" IS NOT NULL AND "status" <> 'REVOKED' AND "deletedAt" IS NULL.
    • MySQL: a VIRTUAL generated column liveLineGrantKey (tenantId:orderLineId:kind while the row is a live line grant, otherwise NULL) plus a unique index on it. Same approach as 1791000000793 in the order package.
    • Fail-safe: if duplicate live rights already exist, it creates nothing, changes no row and logs the affected order lines. It never deletes or updates.
    • Every statement is guarded, and down() drops exactly what up() created, on all three dialects.

  • entitlement/entitlement.service.ts: grant() writes through insertGrant / writeGrant. On a unique violation (kernel isUniqueViolation: 23505, ER_DUP_ENTRY, SQLITE_CONSTRAINT_UNIQUE, or the driver message) it re-reads:

    • a live right for the provenance exists → it is returned as a replay: no event, no key, no conditions;
    • the number was derived from the provenance → the next revision is tried, at most 10 attempts;
    • otherwise → the original error is rethrown. A number the caller stated or the series allocated is a real conflict.
    • Any other error is rethrown untouched.

  • entitlement-number.ts (new): deriveEntitlementNumber(input, revision), moved out of the service. Revision 0 is byte-identical to the old deriveNumber.

  • entitlement.types.ts: ENTITLEMENT_ORDER_STATE_PORT, IEntitlementOrderStatePort, IEntitlementOrderState (status, payment status, completedAt, canceledAt, lines) and IEntitlementOrderLineState (id, quantity, refundedQuantity).

  • events/entitlement-order-settlement.ts (new): pure rules and the status lists they decide on.

    • orderWithdrawalReason / isOrderWithdrawn: the order is CANCELED (or has canceledAt) → ORDER_CANCELED; the payment is REFUNDED → REFUNDED.
    • isOrderSettled: false when the order is withdrawn or its payment is CANCELED. Otherwise true for COMPLETED, completedAt or payment CAPTURED. An ARCHIVED order is decided by its timestamps only.
    • isOrderLinePaidInFull / activatableOrderLines: on a settled order, only the lines with something bought and nothing refunded; a line the port does not list is left out. Without line state, nothing is activated on PARTIALLY_REFUNDED, and every line is activated otherwise.
    • A partly refunded line is left PENDING rather than lowered. The refund handler subtracts what it is told; it does not reconcile to a total. Lowering the right here would therefore apply a refund a second time when that refund's own delivery, possibly still queued, arrives.

  • entitlement/entitlement.service.ts: activateGranted takes an optional orderLineId that narrows an order (equality filter, the same transaction and events).

  • entitlement/entitlement.service.ts: reduce answers a right that is already REVOKED as it stands, with no write and no REDUCED event, decided under the row lock (as revoke, and a reduction to zero, already did). Before, a reduction rewrote a revoked row's quantity and announced it; a REST or GraphQL reduction of a revoked right now returns it unchanged instead.

  • entitlement/entitlement.service.ts: findGranted({ orderId | orderLineId | subscriptionId }, scope) reads the rights one provenance granted, in any status, under the scope stated. It goes through the service's own context-safe read path: the TypeORM repository under TypeORM (the same find the consumer made before, since a delivery has no request user to scope by), and a fresh MikroORM fork under MikroORM. With no provenance named it answers [].

  • events/entitlement-grant.consumer.ts: optional @Inject(ENTITLEMENT_ORDER_STATE_PORT). For a placement with at least one granting line:

    1. It reads the order before granting. If the order is withdrawn, it grants nothing and logs a warning with the reason. It also finishes the idempotent withdrawal of anything an earlier attempt granted, so a retry after a part-way withdrawal does not leave rights in place.
    2. It grants as before.
    3. It reads the order again after granting:
      • If the order is now withdrawn (a cancellation landed while it was granting), it withdraws what the order granted, with the reason that event gives.
      • Otherwise it calls activateGranted({ orderId, orderLineId }) for each admitted line.

    A failed read fails the delivery, so the delivery is retried. The withdrawal loops of order.canceled and the full refund now share one withdrawOrder helper.

    A partial refund on a line granted again after a revocation now lowers only the live right: the revoked one comes out unchanged through reduce's own check. Withdrawal replays still see every row.

    Every read of an event's rights (order, order line, subscription) now goes through findGranted. The consumer used the CRUD find, which under MikroORM reads on the global entity manager; outbox deliveries create no MikroORM request context, and allowGlobalContext is only forced on under TypeORM, so those reads could fail under DB_ORM=mikro-orm. Under TypeORM the statement is unchanged.

  • README.md: the replay rule and the late-placement rules.

@gauzy/plugin-order

  • order/order-state.reader.ts (new): OrderStateReader.findState(orderId, scope) returns { id, status, paymentStatus, fulfillmentStatus, completedAt, canceledAt, lines: [{ id, quantity, refundedQuantity }] } or null.
    • The tenant is required; without one it answers null and never reads across tenants. The organization narrows the read when given, and the lines are read under the same scope.
    • lines is null unless the lines' refundedAmount adds up (exact decimal sum) to at least the order's ledger refundedTotal. The payment package reports line refunds after the refund commits and only logs a lost report, and a refund need not name a line. A short register would understate a refund, so the caller is told it cannot decide per line.
    • The order and its lines are read through the configured ORM inside one OrderUnitOfWork, so it works under MikroORM without a request context.
    • Provided and exported by OrderModule, and exported from the package index.

apps/api

  • plugin-composition.ts: { provide: ENTITLEMENT_ORDER_STATE_PORT, useExisting: OrderStateReader }, added to the exports.

No new dependency, no removed API, endpoint, file or setting. deriveNumber was a private method; its logic moved to the exported helper unchanged.

Flags/settings and their defaults

  • No new runtime flag. B is gated by the port binding. Unbind ENTITLEMENT_ORDER_STATE_PORT in apps/api/src/plugin-composition.ts and the consumer behaves exactly as before: it grants PENDING and reads nothing. Any other host that composes the plugins without binding the port keeps today's behaviour.
  • With the port bound, B does only what the missed events would have done had they come in order:
    • it puts into force the rights that the settlement event would have activated, and only for lines still paid for;
    • it does not grant for an order that a cancellation or full refund would already have withdrawn.
  • A is the database stating the invariant grant() already enforced by reading: one live right per line and kind. No supported path writes a second one on purpose.
  • FEATURE_ENTITLEMENT (default off) still gates every route, as before.

Tests

Node 26, from the worktree root, with NODE_OPTIONS=--experimental-vm-modules:

npx jest -c packages/plugins/entitlement/jest.config.ts
→ Test Suites: 21 passed, 21 total; Tests: 475 passed, 475 total   (develop baseline: 17 suites, 383 tests)

npx jest -c packages/plugins/entitlement/jest.config.ts packages/plugins/entitlement/src/lib/entitlement-lifecycle.dual-orm.spec.ts
→ Test Suites: 1 passed; Tests: 47 passed   (real SQLite, TypeORM and MikroORM halves)

npx jest -c packages/plugins/entitlement/jest.config.ts src/lib/events
→ Test Suites: 4 passed; Tests: 52 passed

npx jest -c packages/plugins/order/jest.config.ts packages/plugins/order/src/lib/order/order-state.reader.spec.ts packages/plugins/order/src/lib/order-totals
→ Test Suites: 3 passed; Tests: 73 passed

npx tsc -p packages/plugins/entitlement/tsconfig.lib.json --noEmit    → clean
npx tsc -p packages/plugins/entitlement/tsconfig.spec.json --noEmit   → clean
npx tsc -p packages/plugins/order/tsconfig.lib.json --noEmit          → clean
npx tsc -p apps/api/tsconfig.app.json --noEmit                      → clean
npx eslint <touched files>  → 0 errors in the lines this PR adds (the one error reported, `prefer-const` on `let manager` in entitlement.service.spec.ts, is on develop already)
prettier --check            → new files clean; in the pre-existing files only the hunks this PR touches were formatted (develop's copies are not prettier-clean, so reformatting whole files would add unrelated churn)

What the new specs cover:

  • Migration (unit, all three dialects): the exact DDL per dialect, catalogue probes, reuse of a MySQL column left behind by an interrupted run, no DDL when the index is present, the table absent or duplicates exist (never a DELETE/UPDATE), the duplicate report, down(), an unsupported dialect, run order.
  • Real SQLite on both TypeORM and MikroORM (entitlement-lifecycle.dual-orm.spec.ts; MikroORM with allowGlobalContext: false):
    • the migration's own up/down;
    • the database refuses a second live right for a line and kind, and admits one after revocation or of another kind;
    • a grant that lost the race returns the winner's right, writing no row, event or key;
    • a derived number after revocation becomes E-…-L-2;
    • order.completed then order.placed through the real consumer ends ACTIVE, and stays PENDING when the order has not settled or no port is bound;
    • on an order refunded in part, only the line still paid for ends ACTIVE, and the refunded line stays PENDING. Without line state, both stay PENDING;
    • on a line that bought 2 with 1 already recorded as refunded, the refund delivered after the placement leaves the right PENDING with quantity 1, so the refund is applied once;
    • a cancelled order (order.canceled delivered first) or a fully refunded order gets no right at all;
    • a partial refund on a line granted again after a revocation lowers only the live right; the revoked row keeps its quantity and gets no REDUCED event;
    • all of these run the consumer's real findGranted read on both ORMs, so the MikroORM half reads rights inside a delivery without a request context.
  • grant() unit: replay for the Postgres (driverError.code 23505), MySQL (ER_DUP_ENTRY), SQLite (SQLITE_CONSTRAINT_UNIQUE) and MikroORM (message only) error shapes; non-unique errors and caller- or series-number conflicts rethrown; revision numbering and its bound.
  • Settlement rules: withdrawal reasons; an ARCHIVED order read by its timestamps; the paid-in-full check, including numeric strings and values that are not numbers; per-line activation with and without line state; deduplication.
  • Consumer:
    • per-line activateGranted calls;
    • the partial refund, with and without line state: a partly refunded line is neither lowered nor activated;
    • a replay never lowers a right raised since;
    • no grant, with a warning, for CANCELED, REFUNDED and archived-cancelled orders;
    • withdrawal when the order is withdrawn between the two reads;
    • read failures before and after granting;
    • no reads when the port is unbound or nothing grants.
  • activateGranted: narrowing to one line of an order; a line given without its order is ignored.
  • findGranted: by order, by line and by subscription, in any status; another tenant's scope reads nothing; no provenance reads nothing.
  • reduce: a revoked right is answered unchanged, with no event, whether or not the quantity asked is lower.
  • OrderStateReader:
    • lines and timestamps on both ORM arms, read in one MikroORM fork;
    • lines: null for a lost line report or a refund on no line;
    • an exact decimal match, and a register ahead of the ledger;
    • tenant required; both reads propagate errors.
  • Number helper.

Jest prints "A worker process has failed to exit gracefully" for suites that import @gauzy/contracts enum values. --detectOpenHandles (in band) reports no open handle and the warning goes away, so it comes from worker teardown, not a leak.

Manual verification still needed

  • The migration has not been run against real PostgreSQL or MySQL here. The statements are asserted per dialect, and SQLite runs them for real on both ORMs. The MySQL key column is VIRTUAL (no table rewrite, same as 1791000000793). The lane plan said STORED; InnoDB supports unique secondary indexes on virtual generated columns, but this is worth one real MySQL run.
  • No end-to-end run with the real outbox dispatcher delivering events out of order. The consumer is exercised directly with the real services on SQLite.

Risk and rollback

  • Existing duplicates: an install that already holds two live rights for one line and kind keeps them, and the index is not created there. The migration names the affected order lines. Once an operator has revoked the extra rights, the logged statements create the index, and grant() keeps its read-before-insert until then.
  • payment.captured has no producer on develop. order.completed is the settlement event in practice, and it is atomic with status = COMPLETED. A future payment.captured producer should commit paymentStatus = CAPTURED before or with the event. Otherwise a placement handled in between reads "not settled" and leaves the rights PENDING, which is today's behaviour, not a regression.
  • Late placement on a refunded line: the line's right is still granted PENDING, as the review asked: it is left PENDING, not skipped. A later full refund event, if any, revokes it as before.
  • Partly refunded line: in the reordered case it stays PENDING, which is develop's behaviour for every line, rather than activating its remaining seats. Activating the remainder safely needs the refund handler to reconcile to totals (durable per-refund tracking); that is a possible follow-up.
  • Refund register lag: if both the ledger mirror and the line mirror of a refund were lost, the order reads as unrefunded. The late placement then activates what an in-order order.completed would have activated on develop. recomputeRefundCounters is the convergence path.
  • Two reads per granting placement while the port is bound: one before granting and one after. They are primary-key and orderId-indexed reads.
  • Subscription grants: subscription-provenance grants are not covered by the new index; their duplicate key is subscription + product + variant.
  • Rollback: revert the PR (down() drops the index, and the MySQL column). To switch off B alone, remove the one binding in plugin-composition.ts.

Owner answers (applied)

  1. Option 2 through the order row and an order-package port: confirmed.
  2. Order already CANCELED or fully REFUNDED at late placement: the placement skips the grant and logs it (consumer and dual-ORM specs). The read after granting also closes the race with a cancellation that lands while it is granting.
  3. MySQL key column: kept VIRTUAL, consistent with the order package.

Review follow-up:

  • Round 1: Greptile and Copilot flagged that a partial refund would activate the refunded line's right. Activation is now decided per line from the order port's line state. Without line state, nothing is activated on PARTIALLY_REFUNDED. Copilot also noted that ARCHIVED hides completion or cancellation, so completedAt and canceledAt are now exposed and used.
  • Round 2 (Copilot):
    • a retry finishes a part-way withdrawal;
    • the line register is trusted only while it accounts for the ledger's refunded total;
    • partly refunded lines are no longer activated at full quantity.
  • Round 3 (Copilot): lowering a partly refunded line's right double-applied a refund whose delivery is still queued, and could undo an operator's extension on a retry. Such a line now stays PENDING, and nothing is ever lowered by the placement.
  • Round 4 (Copilot overview): the consumer's reads of an event's rights went through the CRUD find, which needs a MikroORM request context that deliveries do not have. They now go through EntitlementService.findGranted, the service's context-safe read, on both ORMs.
  • Round 5 (CodeRabbit):
    • a partial refund no longer lowers a revoked right (fixed; this also happened on develop);
    • a reduction now checks the status under the row lock, so a revocation that commits after the refund handler's read is honoured (fixed in reduce, the one place the rule lives);
    • the migration still logs and skips, rather than failing, when duplicate live grants exist. Failing would block the upgrade of an install with legacy duplicates (answered in a PR comment);
    • a settlement delivered after a late placement can still activate a line left PENDING for a refund. That happens on develop too, and the PR does not make it worse. Withholding such lines at settlement would also hold back rights a refund already reduced correctly, so it is left for the refund-reconciliation follow-up (answered in a PR comment).

Closes #11023

🤖 Generated with Claude Code