supervisor: retain lifetime restart history across operator resets (closes #70)

Merged
IC
3mo ago
+173-9

Closes #70.

ck module status reports restarts N/M — the live restart budget. Three operator verbs zero that counter:

supervise.rs   restart_count += 1        crash / respawn paths (3 sites)
supervise.rs   reset_restart_count(...)  operator restart · reload · enable (3 sites)

So restarts 0/3 means either "this module has never crashed" or "this module crashed twice and an operator restarted it". Those are opposite operational situations — one is healthy, one has been failing and was papered over — and the status line renders them identically. An instrument that reports the same value for a healthy module and a repeatedly-failing one is not measuring what its label claims.

Change

A second counter, lifetime_restarts, incremented at every site where restart_count is incremented and never reset. reset_restart_count does not touch it.

Rendering shows it only when it diverges from the budget count, so the healthy case stays unchanged:

restarts 0/3               never crashed
restarts 0/3 (2 lifetime)  budget was reset; crashed twice historically

Budget semantics are deliberately untouched. The reset is correct — an operator restart legitimately grants a fresh budget, and set_enabled(true) revival depends on it. This adds an honest second counter; it does not change when a module gives up.

Wire representation

Option<u32> with serde(default, skip_serializing_if = "Option::is_none").

The Option is load-bearing rather than stylistic. My earlier PR #8 used skip_serializing_if = "is_zero" on a bare u32 in this same struct, which collapsed "zero restarts" and "field absent" into one wire state — the absent-vs-empty collapse that issue #12 was filed about, and it was right to be rejected. Here Some(0) ("known: never restarted") stays distinct from None ("old peer, unknown"), so an old payload keeps its honest unknown instead of asserting zero. Existing golden fixtures are unchanged — they serialize None and skip the field.

Tests

Four arms, each proved load-bearing by breaking the seam it guards:

ArmRed proof
crash twice → operator reset → restart_count 0, lifetime_restarts 2reset mutated to clear lifetime → predicate never satisfied, last state showed lifetime reset
each of the 3 increment sites bumps lifetimehealth site → left 0 right 1; reload site → left 0 right 1; crash site → lifetime stuck at 0 while restart_count advanced
serde round-trip + old payload → Noneskip mutated → encoded null instead of 5
render suppressed vs shownrender mutated → 0/3 instead of 0/3 (2 lifetime)

The per-site arm is the one worth having: a test that only exercises the common crash path stays green if an increment site is missed, and the counter then under-reports silently forever — which would reproduce the exact defect this fixes.

Gates

cargo test --workspace       769 passed  0 failed  1 ignored   (master baseline 765 + 4 new)
cargo clippy --workspace --all-targets -- -D warnings   PASS
cargo fmt --all --check                                 PASS
cargo build --workspace --locked                        PASS
check-wire-crate-versions.sh origin/master              PASS

Baseline measured independently on 2a0fbb67 rather than taken on trust: 765 passed there, 769 here, difference exactly the four new tests.

Additive wire change — subc-control 0.7.0 with the dependent cascade.


View with [code]smith Autofix with [code]smith
Need help on this PR? Tag @codesmith-bot with what you need. Autofix is disabled.


Summary by cubic

Fixes ck module status so the restart budget no longer hides repeated crashes behind an operator reset, distinguishing "never crashed" from "crashed twice, then reset" (closes #70).

  • Adds lifetime_restarts, incremented wherever restart_count increments but never reset, so reset history renders as 0/3 (2 lifetime) while a fresh budget keeps the plain 0/3.
  • Leaves budget reset semantics unchanged; an operator restart legitimately grants a fresh budget, and the lifetime counter does not alter when a module gives up.
  • Adds tests covering all three increment sites, the wire round-trip (old peers stay None), and both render branches so a missed site can't silently under-report.

Wire change

  • SupervisorEntry gains optional lifetime_restarts; Some(0) stays distinct from None on old peers, so unknown history stays honest.
  • Bumps subc-control to 0.7.0, cascading to subc-client-rs 0.8.1 and subc-core 0.8.2.