fix(membership): retain recently-declared-dead entries during pruning

Merged
RE
3mo ago
+73-4

Problem

We recently added MaxDefunctSiloEntries to cap the number of non-active (e.g. Dead) entries kept in the membership table. When the cap is exceeded, the cleanup agent prunes the oldest defunct entries, keeping the most-recently-updated ones so they still appear in membership snapshots.

Recency was measured by EffectiveIAmAliveTime (the later of StartTime and IAmAliveTime). The problem is that declaring another silo dead does not update that silo's IAmAliveTime — only a silo's own periodic self-update does. So a silo that was declared Dead just now, but last reported itself alive long ago, looks like the oldest defunct entry and is pruned immediately, disappearing from snapshots right after it died.

Solution

Reference the entry's suspect times during pruning instead of bumping IAmAliveTime on death. A new internal MembershipEntry.EffectiveUpdateTime returns the later of EffectiveIAmAliveTime and the most recent SuspectTimes vote. Declaring a silo dead records a suspect vote, so a recently-declared-dead entry is now correctly treated as the most-recently-updated defunct entry and is retained.

EffectiveUpdateTime is used consistently in:

  • the cleanup agent's defunct-entry selection (DefunctSiloEntryPriority) and cutoff computation, and
  • the in-memory membership providers' removal predicate (used by the default/dev SystemTarget clustering).

This keeps the death-declaration timestamp as the source of truth without mutating IAmAliveTime.

Notes / scope

  • Cloud/ADO.NET providers are unchanged. The Azure table provider already keys removal off the row write timestamp (which reflects the death write), so recently-dead entries are retained there. The ADO.NET provider keys removal off the IAmAliveTime column and cannot see suspect times without a schema change — that remains a pre-existing limitation and is out of scope here.

Tests

Added MembershipTableCleanupAgent_ThresholdCleanup_RetainsRecentlySuspectedEntries, which sets up two Dead entries where the one with the oldest IAmAliveTime has a very recent suspect vote, and asserts it is the entry that survives pruning. Full membership suite passes (67 tests).