fix(conformance): re-probe runtime-dependent variants on a container-backed server

Merged
VI
12h ago
+303-13

Summary

The conformance sweep runs its fakecloud server with containers disabled, so thousands of probe variants never start real databases, functions or instances. RDS CreateDBInstance refuses without a runtime (InsufficientDBInstanceCapacity, the #2107 fix) instead of reporting an instance with no database behind it, so its two documented-example variants (example_0_To_create_a_DB_instance, examples_diff_0_To_create_a_DB_instance) could never pass: RDS sat at 5537/5539.

Rather than faking a create or carving out an exception, the probe now runs those variants where a runtime exists:

  • Any variant that fails with the shared runtime-required error (fakecloud_core::container_net::CONTAINER_RUNTIME_HINT) is re-probed on its own fresh server with Docker/Podman enabled. One server per variant keeps examples that create the same identifier from colliding.
  • That server is stopped with SIGTERM so it runs every runtime's stop_all() and removes what it started (killed only after a 60s grace period).
  • Only when the sweep owns its server and FAKECLOUD_CONTAINER_CLI is unset (a pinned CLI already chose the runtime). Without a runtime the variants stay failures, with a note explaining why.
  • Only failing variants are re-probed, so the number of containers a sweep can start stays bounded.

The conformance-check CI job runs on ubuntu-latest with Docker and does not set FAKECLOUD_CONTAINER_CLI, so it picks this up. The RDS baseline count is re-recorded in the upcoming baseline batch.

Surfaces: conformance README documents the behavior. No server, API, SDK or website change.

Test plan

  • Unit tests: runtime_required_failures_are_reprobed, reprobe_detects_a_runtime_only_when_the_operator_pinned_none (plus the existing sweep tests)
  • fakecloud-conformance run --services rds with no runtime: 95/97 CreateDBInstance plus the note
  • Same with a container CLI on PATH: both examples re-probed, CreateDBInstance 97/97, RDS 164/164 ops fully passing
  • clippy + fmt clean

Summary by cubic

Fixes the conformance sweep so probe variants that require a container runtime are no longer reported as failures. Variants that fail with the shared runtime-required error are re-probed on their own fresh server with Docker or Podman enabled, then shut down gracefully so the containers they started are removed. This unblocks RDS CreateDBInstance's two documented-example variants, which previously sat at 5537/5539.

  • Re-probing only applies when the sweep owns its server and FAKECLOUD_CONTAINER_CLI is unset; otherwise variants stay failures with a note explaining why.
  • Only failing 5xx variants qualify; the runtime hint inside a 2xx body is a real finding and is left alone.
  • One container-backed server per variant prevents examples that create the same identifier from colliding, and a server that fails to start records that variant as failing rather than aborting the sweep.
  • Shutdown waits for in-flight container operations before asking the server to stop, then removes leftover containers and volumes still bearing the server's label.
  • Unit tests cover the new behaviors; the RDS baseline will be re-recorded in the upcoming baseline batch.