feat(runtime): worker validation gates

Closed
SE
21d ago
+1533-13

Second of five stacked PRs for the worker validation gates DEP (ai-dynamo/dynamo#15897). Based on #1.

What

A worker can run user-supplied checks against itself before it reports ready.

DYN_WORKER_VALIDATION_GATES='{
  "smoke":   {"exec": ["python", "/checks/smoke.py"], "timeout_secs": 900},
  "latency": {"url": "http://127.0.0.1:9100/validate", "timeout_secs": 600}
}'
DYN_WORKER_VALIDATION_HOLD=card   # or "health"; default "card"
  • lib/runtime/src/validation/config.rs parses both variables. It fails startup on invalid JSON, a malformed gate name, both or neither of exec/url, an empty command, a missing or zero timeout_secs, unknown fields, a non-loopback or non-http url, or a HOLD that is set without gates or to an unknown value.
  • validation/runner.rs runs the checks:
    • exec runs the command in the worker container with kill_on_drop; exit 0 is a pass, and the last 2 KiB of stderr becomes the failure reason.
    • url POSTs the gate env as JSON. Connection failures and non-2xx responses are retried with backoff until the deadline. A 2xx {"pass": bool, "reason"?} is final.
  • validation/mod.rs holds WorkerValidation, which the DistributedRuntime owns:
    • Gates run once, in parallel, when the first local engine registers. PR 4 moves this trigger to the loopback endpoint.
    • Verdicts are kept in memory only.
    • Release latches, and release hooks run in order before release. PR 3 uses a hook to write held cards; a failed hook is reported as a publish gate failure.
  • /health ANDs the release state and adds a validation object to its body. /live doesn't change.
  • Startup fails if gates are configured without the system status server, or under SystemProbePolicy::RuntimeOnly (sidecar mode).
  • Metrics: dynamo_component_worker_validation_gate_state{gate} (0 undecided, 1 passed, -1 failed) and dynamo_component_worker_validation_gate_duration_seconds{gate}. prometheus_names.py is regenerated.

With the variables unset, worker_validation() is None and every path is unchanged.

Tests

  • Unit tests for every misconfiguration, and for each loopback spelling.
  • exec: pass, fail with the stderr tail, the tail size bound, a missing program, and a hang that fails at the deadline and kills the child.
  • url: pass and fail verdicts, retried until the runner is up, 500 then 503 then a pass, a dropped connection, never up, and a malformed verdict.
  • Lifecycle: release, one failing gate holds, start runs only once, hooks run in order and are refused after release, a failed hook fails closed, shutdown fails running gates, and the gate env vars.
  • Integration (--features integration):
    • /health is 503 with pending while /live is 200, and turns 200 once the engine registers and the gate passes;
    • a failed gate keeps /health 503 with its reason while /live stays 200;
    • gates fail startup without DYN_SYSTEM_PORT.

cargo clippy -p dynamo-runtime --all-targets --features integration -D warnings is clean.

Stack

  1. #1 Split /live from /health
  2. This PR: runtime validation gates
  3. #3 Hold the model card until release
  4. #4 Loopback OpenAI endpoint
  5. #5 Reference check runner, docs and an example DGD

🤖 Generated with Claude Code