subc-daemon: job-object containment for module grandchildren (#109)

Closed
QI
2mo ago
+1369-0

Linked issue

Approved issue: #109

The agreed scope on that issue is the grandchild kill only. The session/host ownership model (maintainer-owned) is untouched.

What this does

Contains a supervised module's whole process tree in a Windows job object, so teardown reaches helpers a direct-child kill cannot.

A supervised module may spawn helpers of its own — the Synapse embedding module spawns a CUDA worker that holds the GPU allocation. Nothing in teardown took them: Child::kill is TerminateProcess scoped to one pid, and Windows has no process group to signal, so a module terminated rather than asked could not close its own pipes and its grandchildren outlived it. A day of restarts accumulated orphans.

This is the Windows arm only. Unix containment is the process group plus child_roster (already on master); the two are platform-disjoint and compose at the spawn site.

Why a job object rather than a tree walk

A job contains by membership, not ancestry, which covers the two cases a taskkill /T walk cannot:

  • a grandchild spawned between the walk and the kill;
  • a grandchild whose parent has already exited and been reparented, so it is no longer in the tree to be found. Those are exactly the orphans that accumulate.

Two properties are load-bearing:

  • JOB_OBJECT_LIMIT_KILL_ON_JOB_CLOSE — dropping the last handle kills every member. That is what makes containment survive a daemon crash, where no code runs to call anything.
  • Breakaway is deliberately not permitted. Leaving JOB_OBJECT_LIMIT_BREAKAWAY_OK unset is what stops a child from escaping with CreateProcess/CREATE_BREAKAWAY_FROM_JOB. A module that could leave the job could leave its grandchildren behind, which is the defect.

The suspend → assign → resume contract

Assignment must happen before the child can run a single instruction, or it could spawn a grandchild that escapes — the defect in smaller form. CreateProcess has no way to place a process in a job atomically short of building the process by hand with PROC_THREAD_ATTRIBUTE_JOB_LIST, so the child is created suspended, assigned, and then resumed.

There is exactly one place that order lives (contain_spawned_child), and the crate root documents it. resume_main_thread reports failure rather than logging it: a suspended process holding a pid with no way to start is a leak, so the caller kills it and fails the spawn.

An assignment failure is not fatal — an uncontained module behaves exactly as it did before, whereas refusing to start one would be a new outage. It is logged at warn because it means a helper could leak.

Ordering at teardown

start_kill terminates the job tree, and it runs at the existing kill site — after the drain budget and after the reaped child, never before. Containment never changes whether a module is reported as stopped: a job failure is logged and the direct-child kill still decides the outcome.

New crate

subc-jobobject follows the subc-cgroup model: a leaf crate, because subc-daemon forbids unsafe code and the Win32 calls need it. Every FFI call and its safety argument is in one file (sys.rs), and the manual Send for the job handle sits on that side of the boundary rather than on the public API. Sync is deliberately not implemented — concurrent TerminateJobObject and Drop on one handle would be a double-close.

Tests

Three, run against the supervisor rather than the crate, because the claim is about teardown: a crate-level test proves a job can reap a tree, not that the daemon's drain path reaches it.

testwhat it defends
teardown_reaps_the_grandchildthe fix: the grandchild is gone after the supervisor's own drain
an_uncontained_grandchild_survives_a_direct_child_killthe defect reproduction from #109
dropping_containment_reaps_the_grandchildcrash durability: the tree dies with no teardown code running

The mutation control was run: with assignment disabled, teardown_reaps_the_grandchild reddens by name —

grandchild 24616 outlived module teardown: the tree was not contained

The fixture is the sanctioned fake-aft-stub, extended with a grandchild mode rather than a new binary. subc-core and subc-daemon are version-bumped for the wire check.

Verification

  • cargo test -p subc-daemon --lib — 299 passed, 0 failed
  • cargo test -p subc-jobobject — 4 passed, 0 failed (+ doc test)
  • cargo clippy --all-targets -- -D warnings clean on both crates
  • cargo fmt --all -- --check clean
  • Rebased onto current master (f193a178)

Limits

  • Windows only. The #[cfg(windows)] arm compiles out elsewhere; the cgroup lane is a separate containment path with its own tests.
  • The direct-child kill is unchanged. Containment is additive; start_kill still calls it, so nothing about existing stop reporting moves.
  • The grandchild fixture is a parked process. It proves containment reaches a real grandchild; it does not exercise a module's own spawn logic.

View with [code]smith Autofix with [code]smith
Need help on this PR? Tag @codesmith-bot with what you need. Autofix is disabled.


Summary by cubic

Contains a supervised module's whole Windows process tree in a job object, so teardown reaches grandchildren a direct-child kill cannot. A module's CUDA worker held the GPU allocation and outlived restarts, accumulating orphans.

  • New subc-jobobject crate; the child is created suspended, assigned, then resumed so nothing runs before it is contained.
  • The job kills its members when the last handle closes (daemon-crash durability) and forbids breakaway, so a child cannot escape containment.
  • KILL_ON_JOB_CLOSE also fires on every ordinary Windows stop (taskkill or scheduler /End, no SIGTERM handler), killing all modules at once instead of letting them run teardown — accepted for now; a real drain-before-exit stop path is the planned fix.
  • A job-assignment failure is not fatal: the module runs uncontained as before, logged at warn; a resume failure kills the child and fails the spawn.
  • Teardown terminates the job at the existing kill site, and the direct-child kill still decides whether the module is reported stopped.
  • Tests run against the supervisor: teardown reaps the grandchild, an uncontained grandchild survives the same kill (issue #109 reproduction), and closing the job handle reaps the tree with no teardown code running.
  • Windows only; subc-core and subc-daemon are version-bumped.