fix(directory): support rolling upgrades to distributed directory

Merged
RE
3mo ago
+1621-36

Live clusters need to migrate from LocalGrainDirectory to DistributedGrainDirectory without directory stalls, stale routing, or disruptive cache resets during a rolling deployment.

This change adds a realistic three-silo rolling-upgrade suite which keeps sustained hot and fresh grain traffic running while silos are restarted one at a time. It captures phase-aware membership, directory handoff, activation, cache, caller, and error diagnostics, and verifies mixed-mode convergence, unique activations, targeted invalidation, and the absence of whole-cache clears.

The runtime changes:

  • reject requests arriving at a stopping silo with targeted cache invalidation while allowing already-produced responses to complete
  • bound recovery calls to departing members using one shared shutdown-linked cancellation token per silo
  • publish member-token snapshots from the existing membership loop so the invocation hot path avoids per-call cancellation sources, membership observers, and Task.WhenAny
  • preserve ShuttingDown as callable and cancel member operations only at Stopping, Dead, or removal

A bounded retry is used only for the unavoidable in-flight disruption when the client's gateway itself restarts. Persistent failures and all post-convergence failures remain test failures.

Microsoft Reviewers: Open in CodeFlow