fix(runtime): fault outstanding callbacks when a host shuts down

Merged
RE
3mo ago
+191-0

Problem

When an Orleans host (silo or client) shuts down, it can no longer receive responses to grain calls. Any requests still outstanding at that point never complete, so the awaiting caller hangs forever.

This is especially damaging during an ungraceful silo shutdown. If an activation is awaiting an outbound response, it cannot be deactivated, which blocks disposal of its grain context and, in turn, blocks disposal of the host itself. This manifests as a hang during host/ServiceProvider disposal (observed while stopping in-process test clusters, and surfaced by the leak fix in #10272).

Regular activations only cancel their pending operations during disposal, but a [StatelessWorker] grain context awaits worker.Deactivated, which cannot complete while the worker is blocked on an in-flight outbound call whose callback will never be signalled.

Solution

Fault all outstanding grain-call callbacks during host shutdown so in-flight calls observe a terminal SiloUnavailableException instead of hanging:

  • CallbackData.OnHostShutdown() completes the callback with a terminal exception, guarding against double-completion with the same interlocked compare-exchange used by OnTimeout/OnTargetSiloFail.
  • InsideRuntimeClient breaks its outstanding callbacks at the RuntimeInitialize lifecycle stop stage, which runs last (after messaging has stopped), so faulting is always safe. Graceful shutdowns have already drained their grains, so it is effectively a no-op there; only ungraceful shutdowns leave residual callbacks.
  • OutsideRuntimeClient breaks its outstanding callbacks in StopAsync once messaging has stopped, mirroring the silo path so clients no longer hang on in-flight calls at shutdown.

Includes a deterministic client regression test (HostShutdownCallbackTests) that hangs without the fix and passes with it.

This is the base change; the in-process test cluster disposal/leak fix (#10272) builds on top of it.

Microsoft Reviewers: Open in CodeFlow