fix(streaming): use overflow-safe comparison for cached message sequence numbers

Merged
FE
3mo ago
+127-1

Fixes #10246

Problem

CachedMessageExtensions.Compare compares a cached message's sequence number to a token
by casting the 64-bit difference to int:

? (int)(cachedMessage.SequenceNumber - token.SequenceNumber)

For gaps larger than int.MaxValue the truncated result has an arbitrary sign (or is 0
for gaps that are multiples of 2^32), so the comparison silently returns the wrong
ordering.

Such gaps occur in practice with the in-box memory stream provider:
MemoryStreamQueueGrain seeds its sequence counter with DateTime.UtcNow.Ticks on every
activation, so a queue-grain deactivation/reactivation (silo crash, restart, migration)
moves the numbering forward by far more than 2^31 in a single step (2^31 ticks ≈ 3.6
minutes). When a pulling agent's cache holds messages from both sides of such a jump, the
corrupted comparison produces two failure modes in PooledQueueCache:

  • Silent delivery stall. SetCursor treats new (post-jump) messages as older than an
    idle cursor's pre-jump token ("token too new to be in cache") and parks the cursor as
    Unset; delivery stops for every live subscriber with no exception and no log line.
  • Full-cache replay. Placing a cursor at a post-jump token while the oldest cached
    message is pre-jump can take the "token too old" branch; because the purge-metadata gate
    in that branch uses a correct long comparison, the post-jump token passes it and the
    cursor is silently rewound to the oldest cached message, re-delivering the entire
    retained cache. Each replay pass then dies at the epoch boundary with a
    QueueCacheMissException whose token values are self-contradictory
    (Low < Requested < High), and a token-less re-subscribe starts the next pass. In one
    production incident this produced ~1.7M re-deliveries of a ~34.5k-message cache against
    ~7.2k genuine publishes, plus ~5.3k messages that were never delivered to any
    subscriber during the preceding stall.

Solution

Use long.CompareTo, which is overflow-safe, instead of the truncating cast. This matches
the comparison semantics EventSequenceTokenV2.CompareTo already implements. Since the
memory provider's epoch bases are wall-clock ticks, they are monotonic across
reactivations, so a correct 64-bit comparison orders cross-epoch tokens properly — fixing
both failure modes. The EventIndex tiebreak is left as-is: it is an int - int on small
non-negative batch indices and cannot overflow.

Added regression tests to PooledQueueCacheTests:

  • CompareHandlesSequenceNumberGapsLargerThanIntMaxValue — direct Compare checks across
    gaps that previously flipped sign (both directions) or collapsed to zero, plus the
    equal-sequence-number event-index tiebreak.
  • IdleCursorContinuesAfterLargeSequenceNumberJump — an idle cursor must deliver a
    message published after a > int.MaxValue jump (previously: silent stall,
    TryGetNextMessage returned false forever).
  • CursorNotRewoundToOldestMessageAfterLargeSequenceNumberJump — a cursor placed at a
    post-jump token must deliver only that message (previously: rewound to the oldest cached
    message and replayed the cache).

All three fail without the one-line fix and pass with it.

Microsoft Reviewers: Open in CodeFlow