feat: add bounded low-overhead GPU sampling for Agoge runs (#53)

Draft
RM
1d ago
+1975-22

Summary

Adds a per-run NVML sampler for one Agoge agoge_run_id. start records a capability snapshot and GPU samples until stop or the duration bound. The trainer stays a separate command. This is not a resident monitor.

Refs #53
Linear: RM-1230

What landed

  • tools/f0_gpu_sampler.py: start, stop, and run -- <command> (the bracket stops sampling when the command exits or fails; it does not parse Agoge).
  • Samples are bkl.f0_correlation.v1, stamped with the NVML capability_digest from the existing discovery path. Planned cadence_ms, observed_interval_ns, jitter_ns, and collect_duration_ns are on each row. stop writes a bkl_clock_skew record through tools/f0_clock.py.
  • Missing readings stay null: unavailable, unsupported, permission_denied, transient_failure, device_lost. A real zero stays status: ok.
  • start refuses when free VRAM is under 2048 MiB or compute capability is not 12.0. --allow-low-headroom records the breach instead.
  • Efficiency summaries count permission_denied, transient_failure, and device_lost as invalid so they stay out of peaks and means.
  • Contract: docs/f0-gpu-sampler.md. MiniCPM5 recipe: recipes/f0-gpu-sampler.md.
  • CPU CI runs python3 tools/check_f0_gpu_sampler.py.

Tests

python3 tools/check_f0_gpu_sampler.py — 18 passed. Coverage includes full support, partial support, a missing (unavailable) reading, permission denied, device lost, a legitimate 0% utilization, no device, headroom refusal, overhead over budget, recorded jitter, the duration bound, and process start/stop. Joined fake streams pass tools/check_f0_correlation.py.

Still open on ShipOfTheseus

Live GPU acceptance is not done here. ci-cpu uses the NVML fakes. A person still needs to run recipes/f0-gpu-sampler.md on ShipOfTheseus: one MiniCPM5 canary, the joined Agoge marker stream from agoge-forger#147, and the measured overhead.json from that host. This PR does not claim that run.

tools/gpu_ci_gate.py is in this diff, so the existing kernel GPU validation check is in scope. That smoke is not sampler acceptance.

Cited by: Training Mixture Scientist (Grok Bot).

Open in Web Open in Cursor 

Summary by cubic

Adds a bounded per-run NVML sampler that records device state for one Agoge agoge_run_id as bkl.f0_correlation.v1 JSONL. It runs via start/stop around the trainer (or run -- <command>) and exits on stop, the duration bound, or command exit; it does not launch Agoge, infer training phase, or stay resident.

  • Missing readings stay null (unavailable, unsupported, permission_denied, transient_failure, device_lost); a real zero stays status: ok.
  • start refuses when free VRAM is under 2048 MiB or compute capability is not 12.0; --allow-low-headroom records the breach instead.
  • stop writes a clock-skew record and an overhead summary; efficiency blocks count denied/transient/lost readings as invalid.
  • CPU CI runs tools/check_f0_gpu_sampler.py against NVML fakes (18 scenarios); live GPU acceptance on ShipOfTheseus is documented in a recipe but not executed here.