Resolve TimeProvider via keyed DI per subsystem (fixes FakeTimeProvider timer test CI hang)

Merged
RE
3mo ago
+464-95

Problem

TimerTests.TimerOrleansTest_Parallel and sibling timer tests intermittently hung for ~10 minutes in CI, failing builds.

Root cause: the timer test fixtures install a FakeTimeProvider as the silo's global TimeProvider. FakeTimeProvider.Advance invokes every due timer callback synchronously and inline on the advancing thread. The silo's background maintenance loops (cluster membership, gateway/grain-directory maintenance, activation working-set monitoring, request monitoring, reminders, health checks) also read that same ambient clock. Advancing the fake clock resumed those loops inline; each observed its deadline had passed, immediately re-armed a fresh Task.Delay and ran again — a CPU-bound spin that never returned from Advance, hanging the test until CI's blame-hang timeout.

Solution

Introduce per-area keyed TimeProvider resolution so individual subsystems can be driven by different clocks:

  • TimeProviderNames (new, public) defines one key per area (grains, reminders, messaging, system timers, activation management, grain directory, streaming, transactions, durable jobs, journaling, caching).
  • Each consumer resolves its area's clock via [FromKeyedServices(TimeProviderNames.X)] (or GetKeyedService).
  • The silo/client register a single KeyedService.AnyKey fallback that resolves to the unkeyed default TimeProvider, so every key defaults to the default provider and production behavior is unchanged.

This replaces the earlier test-only IAsyncTimerFactory override. Tests install a FakeTimeProvider as the default provider (grain timers + grain-side delays) and pin all background areas to TimeProvider.System via UseTimeProviderForBackgroundAreas (in Orleans.TestingHost). Advancing the fake clock then only ever fires the grain timers under test and never resumes background loops inline. A fail-fast watchdog around Advance converts any future regression into an immediate, diagnosable failure instead of a multi-minute hang.

Validation

  • TimerOrleansTest (15) + AsyncEnumerableGrainCallTests (50): pass; 20-iteration stress loop of the timer tests: 0 hangs.
  • In-memory reminder tests (ReminderTestClock cascade), journaling (331), durable jobs (27), streaming BVT (64), and transaction overload detector tests: pass.
  • Public API surface regenerated for Orleans.Core and Orleans.TestingHost.