fix(streaming): use overflow-safe comparison for cached message sequence numbers
Fixes #10246
Problem
CachedMessageExtensions.Compare compares a cached message's sequence number to a token
by casting the 64-bit difference to int:
? (int)(cachedMessage.SequenceNumber - token.SequenceNumber)
For gaps larger than int.MaxValue the truncated result has an arbitrary sign (or is 0
for gaps that are multiples of 2^32), so the comparison silently returns the wrong
ordering.
Such gaps occur in practice with the in-box memory stream provider:
MemoryStreamQueueGrain seeds its sequence counter with DateTime.UtcNow.Ticks on every
activation, so a queue-grain deactivation/reactivation (silo crash, restart, migration)
moves the numbering forward by far more than 2^31 in a single step (2^31 ticks ≈ 3.6
minutes). When a pulling agent's cache holds messages from both sides of such a jump, the
corrupted comparison produces two failure modes in PooledQueueCache:
- Silent delivery stall.
SetCursortreats new (post-jump) messages as older than an
idle cursor's pre-jump token ("token too new to be in cache") and parks the cursor as
Unset; delivery stops for every live subscriber with no exception and no log line. - Full-cache replay. Placing a cursor at a post-jump token while the oldest cached
message is pre-jump can take the "token too old" branch; because the purge-metadata gate
in that branch uses a correct long comparison, the post-jump token passes it and the
cursor is silently rewound to the oldest cached message, re-delivering the entire
retained cache. Each replay pass then dies at the epoch boundary with a
QueueCacheMissExceptionwhose token values are self-contradictory
(Low < Requested < High), and a token-less re-subscribe starts the next pass. In one
production incident this produced ~1.7M re-deliveries of a ~34.5k-message cache against
~7.2k genuine publishes, plus ~5.3k messages that were never delivered to any
subscriber during the preceding stall.
Solution
Use long.CompareTo, which is overflow-safe, instead of the truncating cast. This matches
the comparison semantics EventSequenceTokenV2.CompareTo already implements. Since the
memory provider's epoch bases are wall-clock ticks, they are monotonic across
reactivations, so a correct 64-bit comparison orders cross-epoch tokens properly — fixing
both failure modes. The EventIndex tiebreak is left as-is: it is an int - int on small
non-negative batch indices and cannot overflow.
Added regression tests to PooledQueueCacheTests:
CompareHandlesSequenceNumberGapsLargerThanIntMaxValue— directComparechecks across
gaps that previously flipped sign (both directions) or collapsed to zero, plus the
equal-sequence-number event-index tiebreak.IdleCursorContinuesAfterLargeSequenceNumberJump— an idle cursor must deliver a
message published after a > int.MaxValue jump (previously: silent stall,
TryGetNextMessagereturnedfalseforever).CursorNotRewoundToOldestMessageAfterLargeSequenceNumberJump— a cursor placed at a
post-jump token must deliver only that message (previously: rewound to the oldest cached
message and replayed the cache).
All three fail without the one-line fix and pass with it.