#5 feat: reference check runner, docs and example DGD for validation gates

Fifth of five stacked PRs for the worker validation gates DEP (ai-dynamo/dynamo#15897). Based on #4.

What

• dynamo.validation.completion is a built-in check. It sends one chat completion to OPENAI_BASE_URL for DYN_VALIDATION_MODEL. It passes on content, reasoning content or tool calls, and fails with a reason on an HTTP error, no choices, an empty message, or an unreachable endpoint. It can be used directly as an exec gate.
• dynamo.validation.runner is a stateless aiohttp check runner for url gates.
  • POST /validate/<name> runs the command configured with --check NAME=COMMAND (or NAME=builtin:completion) once. Only the gate env keys from the request body are passed to the check.
  • It answers {"pass": true} or {"pass": false, "reason": "<status>: <stderr tail>"}.
  • An unknown check name gets a definite failure verdict, so the gate fails at once instead of retrying until its deadline.
  • On timeout or shutdown it kills the check's whole process group. It binds loopback, has no probes, and never exits on a failed check.
• Docs: docs/fern/pages/kubernetes/fault-tolerance/validation-gates.md (added to the Fault Tolerance nav). It covers the contract, what a check sees, hold modes, verdicts and retries, /health and metrics, sidecar rules, rollout settings (max-surge: 1, max-unavailable: 0), scope and the known bypasses.
• Example: examples/backends/vllm/deploy/agg_validation_gates.yaml, with one exec gate (completion), one url gate served by a check-runner sidecar, and rollout annotations.
• Operator: a comment-only fix to the worker readiness probe in component_worker.go. No operator behaviour or template changes.

Tests

• Unit tests: components/src/dynamo/validation/tests (18 passed).
  • Runner: check parsing, pass/fail/env, a missing program, a timeout that kills the process group, and HTTP verdicts including an unknown check.
  • Completion: three pass shapes, three fail shapes, an unreachable endpoint, and missing env.
• E2E: tests/fault_tolerance/test_worker_validation_gates.py, run on CPU with mocker workers, a real frontend, etcd and NATS. All 3 pass in about 16 s locally.
  • test_failing_gate_holds_the_worker_and_old_worker_keeps_serving: the held worker has /health 503 with the gate's reason and /live 200. It serves 0 of 20 client requests, all of which the old worker serves.
  • test_passing_gates_release_the_worker: the completion gate passes through the loopback endpoint while a second gate still holds. During the hold the model isn't listed and requests fail. On release the card is published and requests are served by the worker.
  • test_a_restarted_worker_is_checked_again: a second worker process runs its gates again.

Not covered here

The DEP's Kubernetes rollout acceptance (a DGD roll with maxUnavailable: 0 stalls on a failing gate and completes when it's flipped) needs a cluster. It's left for a deployment-level test.

Stack

1. #1 Split /live from /health
2. #2 Runtime validation gates
3. #3 Hold the model card until release
4. #4 Loopback OpenAI endpoint
5. This PR: reference check runner, docs and an example DGD

🤖 Generated with Claude Code