supervisor: bounded per-module terminal ring so a missed exit is countable

Merged
IC
2mo ago
+597-20

Closes #12. Built to the shape you specified: timestamped exit records held across respawns, served by their own op, with an eviction count so a returning observer knows its backfill is a suffix.

What landed

  • supervisor.terminals — its own op, not inflating supervisor.list. Same size-discipline argument as supervisor.stderr_tail.
  • TerminalDisposition — Stopped / Disabled / Failed / Restarting, recorded at the point the supervisor chooses it rather than inferred afterwards.
  • dropped on eviction — the field that turns "can't count what I missed" into a number.
  • last_exit_ms on SupervisorEntry regardless, so every existing reader gets ordering without adopting the new op.
  • ck module terminals <id>, rendering the dropped count.
  • In-memory, lost on daemon restart. That boundary holds — durable history stays with the observer layer per #4.

drain_child_to_state also observes terminals

I briefed this at on_child_exit and that was incomplete. drain_child_to_state reaches a terminal on five more paths — restart, reload, health restart, stop, disable — and none of them were in my scope. They record their disposition now too.

Worth stating because it changes what the feature is: had it shipped covering only on_child_exit, the ring would have held crashes while a reload or an operator stop left no record. An observer would then read "no terminals" for a module that had been deliberately cycled, which is the same absence-reads-as-health defect this issue exists to fix. Credit to the implementer for checking rather than taking my list.

The daemon-incarnation signal

The response carries daemon_started_at_ms. An empty ring immediately after daemon start otherwise reads as "nothing has died" when it means "I cannot tell you what died" — reproducing the defect one layer out. With it, an observer can tell a fresh daemon from a quiet one and re-baseline.

Today's OOM is the case that motivated it: three modules died, the successor cold-started, and nothing on the wire said an epoch had passed.

No shared ring core with StderrRing

You asked whether the two rings should share a bounded-ring core. They shouldn't. Of five dimensions, four don't transfer:

stderr ringterminal ring
bounded deque + droppedyesyes
total byte capload-bearing (unbounded line length)meaningless
per-line truncationload-bearingmeaningless
three-state captureload-bearingmeaningless
in-band boundary markerneededthe record is the boundary

Strip those and the shared core is a VecDeque with a length cap and a counter — about fifteen lines. The specific risk is that StderrRing's three-state capture enum gets flattened by a consumer that doesn't need it, and collapsing three world-states into two was the original defect in this area. stderr_tail.rs is untouched here.

The golden fixture is generated through the real handler

It spawns a module, crashes it twice, waits for two terminals, builds a real frame, and runs handle_control_frame — the serialized response is the fixture. Not hand-constructed, for the reason that cost a review round on #10: a fixture built from the same misconception as the implementation goes green while the implementation is wrong.

git diff --stat origin/master -- crates/subc-control/tests/golden/
 catalog_list.json                    | 3 ++-   op string
 server_describe.json                 | 3 ++-   op string
 server_describe_with_counters.json   | 3 ++-   op string
 supervisor_list.json                 | 1 +     last_exit_ms
 supervisor_entry.json                | 1 +     last_exit_ms
 supervisor_terminals.json            | 17 +++  new

Every delta attributable to exactly one cause. Nothing else moved.

Two process notes

I re-introduced the /bin/sh defect and caught it before you did. Two new test spawns used PathBuf::from("/bin/sh") — the exact thing that blocked #10 on Windows. I fixed that PR by extending fake-aft-stub, then wrote this brief without mentioning it. Both now use the stub with FAKE_AFT_EXIT_CODE; neither test needed shell behaviour beyond an exit code. git diff origin/master -- crates/ | grep '^+' | grep /bin/sh returns nothing.

Versions bumped — subc-control 0.2.0, subc-core 0.4.0, Cargo.lock committed. I ran check-wire-crate-versions.sh myself this time rather than discovering it in review.

Gates

cargo test --workspace              31 binaries, 0 failures
cargo fmt --check                   clean
cargo clippy --all-targets          clean
cargo check --workspace --locked    clean
check-wire-crate-versions.sh        6 examined, 0 violations

Both process-spawning tests pass 10/10 in a loop. Windows I can't verify — same ask as #10, whenever the twin is convenient.


View with [code]smith Autofix with [code]smith
Need help on this PR? Tag @codesmith with what you need. Autofix is disabled.


Summary by cubic

Adds a bounded per-module terminal history and exposes it via a new supervisor.terminals op so observers can see and count missed exits. Previously only the latest exit was visible and operator-driven cycles could leave no record; now every terminal exit records its timestamp and the supervisor’s chosen disposition.

  • API: adds supervisor.terminals with daemon_started_at_ms, ordered entries, and a dropped count for evictions; supervisor.list now includes last_exit_ms for ordering without adopting the new op.
  • Coverage: records terminals on crash and on restart, reload, health restart, stop, and disable paths, not just on_child_exit.
  • CLI: adds ck module terminals <id> (JSON or human-readable), showing retained count and dropped.
  • Wire/version: bumps subc-control to 0.2.0 and subc-core to 0.4.0; subc-client-rs now depends on subc-control 0.2.
  • Migration: observers needing exit history should call supervisor.terminals; existing readers can use last_exit_ms immediately.
  • Persistence: terminal history is in-memory and resets on daemon restart; use daemon_started_at_ms to distinguish a fresh daemon from a quiet one.