chore(conformance): fail on any failing variant; re-baseline at 257,288/257,288

Merged
VI
6h ago
+441-385

Summary

Every implemented service now passes every variant on the refreshed AWS models, so the check enforces that.

  • Strict check:
    • conformance check fails on any failing variant and lists each failing operation (service Op: passed/total). An operation with zero variants also fails, since nothing checked it.
    • The per-service ratchet against conformance-baseline.json stays, with no default flake margin; FAKECLOUD_CONFORMANCE_FLAKE_MARGIN can still relax a local run, and CI doesn't set it.
    • The overall ratchet is removed: it could never fire before a per-service one.
  • Startup wait: the sweep server gets 60s to start (a cold debug binary missed the old 5s), matching the container re-probe server.
  • Re-baseline:
    • 257,288/257,288 variants across 7,610 probed operations, regenerated on main after the conformance batches (#2723, #2724, #2728, #2732, #2733, #2734). RDS runtime-dependent variants were re-probed against a Docker-backed server.
    • Totals moved because of the aws-models refresh (new variants) and the #2734 skip of byte-identical empty-trailing-label requests (-99).
  • Doc counts: 249,849 -> 257,288 in README, AGENTS.md, the conformance/parity/what-it-is docs, landing and vs/ pages, llms.txt/llms-full.txt and the home template (blog posts untouched). check-doc-counts.sh passes, and the ops index is unchanged.
  • Repo description: the GitHub description gets synced right after merge.

Test plan

  • cargo +stable clippy -p fakecloud-conformance --all-targets -D warnings, fmt
  • conformance bin tests (112), lib tests (106), including every_failing_operation_is_a_check_failure
  • full local probe on main: 7610/7610 ops, 257288/257288 variants
  • bash scripts/check-doc-counts.sh OK; generate-operations-index.sh no diff
  • CI conformance-check runs the strict check against the new baseline

Summary by cubic

Conformance checks now fail on any failing or unchecked variant, making the 100% pass requirement enforceable in CI. The baseline is refreshed to 257,288 passing variants across 106 services.

  • The per-service baseline ratchet has no default flake margin; FAKECLOUD_CONFORMANCE_FLAKE_MARGIN can relax local runs.
  • The sweep server startup timeout is now 60 seconds to accommodate cold debug builds.
  • Updated published conformance counts to match the refreshed baseline.