fix(runtime): fault outstanding callbacks when a host shuts down
Problem
When an Orleans host (silo or client) shuts down, it can no longer receive responses to grain calls. Any requests still outstanding at that point never complete, so the awaiting caller hangs forever.
This is especially damaging during an ungraceful silo shutdown. If an activation is awaiting an outbound response, it cannot be deactivated, which blocks disposal of its grain context and, in turn, blocks disposal of the host itself. This manifests as a hang during host/ServiceProvider disposal (observed while stopping in-process test clusters, and surfaced by the leak fix in #10272).
Regular activations only cancel their pending operations during disposal, but a [StatelessWorker] grain context awaits worker.Deactivated, which cannot complete while the worker is blocked on an in-flight outbound call whose callback will never be signalled.
Solution
Fault all outstanding grain-call callbacks during host shutdown so in-flight calls observe a terminal SiloUnavailableException instead of hanging:
CallbackData.OnHostShutdown()completes the callback with a terminal exception, guarding against double-completion with the same interlocked compare-exchange used byOnTimeout/OnTargetSiloFail.InsideRuntimeClientbreaks its outstanding callbacks at theRuntimeInitializelifecycle stop stage, which runs last (after messaging has stopped), so faulting is always safe. Graceful shutdowns have already drained their grains, so it is effectively a no-op there; only ungraceful shutdowns leave residual callbacks.OutsideRuntimeClientbreaks its outstanding callbacks inStopAsynconce messaging has stopped, mirroring the silo path so clients no longer hang on in-flight calls at shutdown.
Includes a deterministic client regression test (HostShutdownCallbackTests) that hangs without the fix and passes with it.
This is the base change; the in-process test cluster disposal/leak fix (#10272) builds on top of it.