supervisor: bounded per-module stderr tail, so a crash cause outlives the log
Implements #7, to your spec on that thread. Stderr is piped per child into a bounded in-memory ring the supervisor owns, and every line is forwarded on so the daemon log keeps its current content.
$ ck module stderr claustrum
... 143 earlier line(s) dropped
config error: missing top-level `storage`
--- process start ---
config error: missing top-level `storage`
What's here
The ring (crates/subc-core/src/stderr_tail.rs, new). Three bounds, not two: total lines, total bytes, and per-line truncation — without the third, one enormous line evicts the whole tail while being unreadable itself. Truncation is carried as a truncated boolean so a consumer can branch without string-matching. dropped_lines is reported because a tail that silently starts mid-history reads as a module that never explained itself.
Capture state is typed, per your second design point. Captured and NotCaptured { reason } are distinct, so "the module printed nothing" and "nobody was listening" don't render alike. That's the detail - shape from #3, and reproducing it inside the fix for it would have been a poor outcome.
Restart boundaries are in-band. TailEntry::ProcessStart sits between the lines it separates, because which side of a restart a line falls on is unanswerable from a count. The ring outlives the process — a ring recreated per spawn would be empty exactly when asked. The first boundary is suppressed: one with nothing before it divides nothing, and emitting it would make a silent module render as a lone marker.
The tap forwards (supervise.rs). Your 4727-of-5000 measurement is why this isn't optional — a tap that captured without forwarding would leave the daemon log nearly empty and every existing reader would report clean on nothing. An absence that reads as calm is worse than the interleaving it replaces.
Line atomicity became this daemon's problem, so it's handled here rather than left to emitters. Previously a module's own write reached the fd in one syscall; reading a pipe and re-emitting can split a line that was atomic. The reader reassembles and writes each complete line in a single call, and the test asserts one write per line rather than merely correct bytes. A pending-line ceiling bounds reassembly, because a module writing without newlines must not grow the daemon's memory — a module fault becoming a daemon fault is what supervision exists to prevent.
Stdout stays inherited. Modules use it for ordinary output rather than diagnostics, so piping it would double the reader tasks for no diagnostic gain.
supervisor.stderr_tail is a separate op rather than a field on supervisor.list — per your first design point, the tail is kilobytes per module and list renders every module. Caps ride on the request.
Review
I wrote the first version and it was not good enough to ship. An independent reviewer returned REVISE with four blocking findings, all real:
max_linescounted restart boundaries as lines, so-n 20on a crash-looping module could return a screen of markers and zero output — the exact failure this feature exists to fix, reintroduced inside the fix. All eleven of my tests passed around it; it took a probe to surface.- The op was implemented but never added to
SUBC_CONTROL_OPS, soserver.describeclients couldn't discover it. - Truncation was encoded three times: a marker inside the text, the boolean, and a suffix appended by
ck. cargo fmthad never been run.
Worth flagging one of those specifically, since it's about the fixture rather than the code: my golden fixture hid the truncation bug. I built it by constructing the wire type directly, so it never passed through truncate_line — it encoded what I intended rather than what the code produced, and stayed green while the code was wrong. The added test goes through the real ring→wire path via the control handler. The fixture itself is unchanged across all three commits, which is what makes the fix demonstrably at the source rather than papered over by regenerating it.
A follow-up round closed a config-coherence edge: max_line_bytes > max_bytes let a boundary be evicted before the oversized line it attributed. Fixed by making that config unconstructible — fields are private behind StderrTailConfig::new, which clamps — rather than teaching eviction to tolerate a configuration nobody can sensibly want.
Verification
cargo test --workspace 31 test binaries, 0 failures
cargo clippy --workspace --all-targets no warnings
cargo fmt --check clean
Note for anyone running these on a box where the harness is itself a supervised module: 1b5d8f2's trap is live and cost me an hour. env -u SUBC_MODULE_ID -u SUBC_LAUNCH_NONCE is required or five real_daemon tests fail with bad_consumer_identity against unmodified upstream. Your note in that file is accurate; I read the reds as mine before checking the baseline.
Two things for you
Absent vs empty, resolved opposite ways in two PRs. Here the states are genuinely different and stay distinct. In #8 I collapsed absent to 0 for restart_count because both cases mean no restarts to report. I think that's right in each case rather than inconsistent, but they're adjacent enough that you may want them to match.
An unrelated flake, seen once during this work and not reproduced on retry or a full rerun: a forwarding test got Goodbye where it expected Response. Not touched by this change and I can't reproduce it, so it's a report rather than a claim — but a flake in route teardown ordering seemed worth naming rather than dropping.
Need help on this PR? Tag @codesmith with what you need. Autofix is disabled.
Summary by cubic
Adds bounded per-module stderr capture with restart markers and per-line truncation, exposed via supervisor.stderr_tail and ck module stderr, so crash causes persist beyond log rotation while still forwarding lines to the daemon log. This preserves line atomicity on forward and keeps stdout inherited.
-
New Features
- Per-module stderr ring in
subc-corewith caps: max lines, max bytes, and per-line byte limit; reportsdropped_linesand in-bandProcessStartmarkers. - Typed capture state:
CapturedvsNotCaptured { reason }. - New control op
supervisor.stderr_tailinsubc-controlwith request caps (max_lines,max_bytes) and wire types (StderrTail,StderrTailEntry). - CLI:
ck module stderr <id> [-n <count>]prints tail, shows dropped count and restart markers; supports--json. - Stderr pump reads child pipe, reassembles full lines, forwards each line in a single write to the daemon’s stderr; ring survives restarts; pending-line ceiling prevents unbounded memory; stdout stays inherited.
- Per-module stderr ring in
-
Bug Fixes
max_linescounts only stderr lines (not restart markers).- Advertised
supervisor.stderr_tailinserver.describeand golden tests. - Single source of truncation truth (boolean on entries; no duplicated markers).
- Clamp incoherent configs (
max_line_bytes>max_bytes) at construction to keep boundaries consistent.