supervisor: bounded per-module terminal ring so a missed exit is countable
Closes #12. Built to the shape you specified: timestamped exit records held across respawns, served by their own op, with an eviction count so a returning observer knows its backfill is a suffix.
What landed
supervisor.terminals— its own op, not inflatingsupervisor.list. Same size-discipline argument assupervisor.stderr_tail.TerminalDisposition—Stopped/Disabled/Failed/Restarting, recorded at the point the supervisor chooses it rather than inferred afterwards.droppedon eviction — the field that turns "can't count what I missed" into a number.last_exit_msonSupervisorEntryregardless, so every existing reader gets ordering without adopting the new op.ck module terminals <id>, rendering the dropped count.- In-memory, lost on daemon restart. That boundary holds — durable history stays with the observer layer per #4.
drain_child_to_state also observes terminals
I briefed this at on_child_exit and that was incomplete. drain_child_to_state reaches a terminal on five more paths — restart, reload, health restart, stop, disable — and none of them were in my scope. They record their disposition now too.
Worth stating because it changes what the feature is: had it shipped covering only on_child_exit, the ring would have held crashes while a reload or an operator stop left no record. An observer would then read "no terminals" for a module that had been deliberately cycled, which is the same absence-reads-as-health defect this issue exists to fix. Credit to the implementer for checking rather than taking my list.
The daemon-incarnation signal
The response carries daemon_started_at_ms. An empty ring immediately after daemon start otherwise reads as "nothing has died" when it means "I cannot tell you what died" — reproducing the defect one layer out. With it, an observer can tell a fresh daemon from a quiet one and re-baseline.
Today's OOM is the case that motivated it: three modules died, the successor cold-started, and nothing on the wire said an epoch had passed.
No shared ring core with StderrRing
You asked whether the two rings should share a bounded-ring core. They shouldn't. Of five dimensions, four don't transfer:
| stderr ring | terminal ring | |
|---|---|---|
bounded deque + dropped | yes | yes |
| total byte cap | load-bearing (unbounded line length) | meaningless |
| per-line truncation | load-bearing | meaningless |
| three-state capture | load-bearing | meaningless |
| in-band boundary marker | needed | the record is the boundary |
Strip those and the shared core is a VecDeque with a length cap and a counter — about fifteen lines. The specific risk is that StderrRing's three-state capture enum gets flattened by a consumer that doesn't need it, and collapsing three world-states into two was the original defect in this area. stderr_tail.rs is untouched here.
The golden fixture is generated through the real handler
It spawns a module, crashes it twice, waits for two terminals, builds a real frame, and runs handle_control_frame — the serialized response is the fixture. Not hand-constructed, for the reason that cost a review round on #10: a fixture built from the same misconception as the implementation goes green while the implementation is wrong.
git diff --stat origin/master -- crates/subc-control/tests/golden/
catalog_list.json | 3 ++- op string
server_describe.json | 3 ++- op string
server_describe_with_counters.json | 3 ++- op string
supervisor_list.json | 1 + last_exit_ms
supervisor_entry.json | 1 + last_exit_ms
supervisor_terminals.json | 17 +++ new
Every delta attributable to exactly one cause. Nothing else moved.
Two process notes
I re-introduced the /bin/sh defect and caught it before you did. Two new test spawns used PathBuf::from("/bin/sh") — the exact thing that blocked #10 on Windows. I fixed that PR by extending fake-aft-stub, then wrote this brief without mentioning it. Both now use the stub with FAKE_AFT_EXIT_CODE; neither test needed shell behaviour beyond an exit code. git diff origin/master -- crates/ | grep '^+' | grep /bin/sh returns nothing.
Versions bumped — subc-control 0.2.0, subc-core 0.4.0, Cargo.lock committed. I ran check-wire-crate-versions.sh myself this time rather than discovering it in review.
Gates
cargo test --workspace 31 binaries, 0 failures
cargo fmt --check clean
cargo clippy --all-targets clean
cargo check --workspace --locked clean
check-wire-crate-versions.sh 6 examined, 0 violations
Both process-spawning tests pass 10/10 in a loop. Windows I can't verify — same ask as #10, whenever the twin is convenient.
Need help on this PR? Tag @codesmith with what you need. Autofix is disabled.
Summary by cubic
Adds a bounded per-module terminal history and exposes it via a new supervisor.terminals op so observers can see and count missed exits. Previously only the latest exit was visible and operator-driven cycles could leave no record; now every terminal exit records its timestamp and the supervisor’s chosen disposition.
- API: adds
supervisor.terminalswithdaemon_started_at_ms, ordered entries, and adroppedcount for evictions;supervisor.listnow includeslast_exit_msfor ordering without adopting the new op. - Coverage: records terminals on crash and on restart, reload, health restart, stop, and disable paths, not just
on_child_exit. - CLI: adds
ck module terminals <id>(JSON or human-readable), showing retained count anddropped. - Wire/version: bumps
subc-controlto0.2.0andsubc-coreto0.4.0;subc-client-rsnow depends onsubc-control0.2. - Migration: observers needing exit history should call
supervisor.terminals; existing readers can uselast_exit_msimmediately. - Persistence: terminal history is in-memory and resets on daemon restart; use
daemon_started_at_msto distinguish a fresh daemon from a quiet one.