fix(directory): support rolling upgrades to distributed directory
Live clusters need to migrate from LocalGrainDirectory to DistributedGrainDirectory without directory stalls, stale routing, or disruptive cache resets during a rolling deployment.
This change adds a realistic three-silo rolling-upgrade suite which keeps sustained hot and fresh grain traffic running while silos are restarted one at a time. It captures phase-aware membership, directory handoff, activation, cache, caller, and error diagnostics, and verifies mixed-mode convergence, unique activations, targeted invalidation, and the absence of whole-cache clears.
The runtime changes:
- reject requests arriving at a stopping silo with targeted cache invalidation while allowing already-produced responses to complete
- bound recovery calls to departing members using one shared shutdown-linked cancellation token per silo
- publish member-token snapshots from the existing membership loop so the invocation hot path avoids per-call cancellation sources, membership observers, and
Task.WhenAny - preserve
ShuttingDownas callable and cancel member operations only atStopping,Dead, or removal
A bounded retry is used only for the unavoidable in-flight disruption when the client's gateway itself restarts. Persistent failures and all post-convergence failures remain test failures.