test(reminders): enable provider failover coverage

Merged
RE
2mo ago
+355-54

Closes #4319

Problem

The Azure Table and Cosmos generic-type reminder failover tests remained skipped because their original worker tasks kept calling grains while a silo was shutting down. The tests also raced reminder-service startup, membership convergence, and ownership reconciliation, then left joined silos in their shared fixtures.

Solution

Move the scenario into ReminderTestsBase with one fake-time driver and explicit startup, ownership, schedule, tick, and counter barriers. Add a fail-closed TestingHost barrier which requires the expected membership, gateway, grain-directory, and manifest views before reminder reconciliation; a timeout throws before any grain call runs. After convergence, invoke each grain exactly once and propagate any rejection. Select a reminder actually owned by the disposable joined silo, replace that silo during failover, and restore the exact baseline topology in finally. Enable the Azure Table and Cosmos cases, add the in-memory equivalent, and cover rejection and timeout propagation directly.

Rationale

This preserves the intended one-failure/one-join topology while removing shutdown-call races without hiding the original failure mode. The scenario validates real ownership recovery across both grain implementations without weakening assertions or contaminating later fixture tests.

Microsoft Reviewers: Open in CodeFlow