feat: optional signal propagation to running DAG processes

Merged
LO
24d ago
+2766-184

Summary

Implements #568: opt-in shutdown signal propagation to runs launched by dagu server, dagu scheduler, and dagu start-all, including API/UI and queued runs. Disabled by default.

  • Unix supervisors forward the original SIGINT or SIGTERM to each registered run's process group and wait for the run to exit after cleanup and terminal status persistence. Windows uses the existing forced process-tree termination adapter.
  • Each run retains its first termination request and one cleanup deadline for workflow steps, repeated iterations, and retry/repeat/delay waits. Concurrent stop requests join that deadline; canceling a stop caller does not shorten it.
  • max_clean_up_time_sec excludes lifecycle handlers. Forced cleanup stops steps without canceling handlers or skipping eligible terminal handlers. Init and terminal handlers retain their own timeout_sec and caller-context cancellation. Supervisors still wait for handlers; a second shutdown signal can force supervisor exit.
  • Termination before execution aborts startup and persists Aborted. Factories and socket readiness observe cancellation. Environment, secret, and profile resolution use an isolated DAG snapshot, so a provider that ignores cancellation cannot restart execution or overwrite an aborted run with late results or panics. Startup interrupted before execution skips lifecycle handlers.
  • Coordinator attempt rejection initiates termination without blocking the status consumer, allowing progress reporting and eligible handlers to finish.
  • Graceful resends preserve step cleanup, including standalone runs with propagation disabled. Canceled retry backoff retains the last executor error while keeping the step aborted. Startup panics log the original provider stack before propagating the original panic value.
  • Scheduler SIGHUP/SIGQUIT, ordinary supervisor context cancellation, and HA lock loss leave runs alive for adoption. Distributed worker runs retain their existing signal propagation behavior.
  • Configure with signal_handling.enable_propagation or DAGU_SIGNAL_PROPAGATION. Explicit canonical/environment values, including false, override the legacy signalHandling.enablePropagation setting accepted in admin.yaml.

Process launch and registry admission share the shutdown boundary. Runs started after the propagation snapshot are excluded. Provider calls that ignore cancellation may continue until their I/O returns in the embedded engine; their results remain isolated from the aborted run.

Validation

  • Handler and diagnostic regressions failed before their fixes and pass afterward.
  • The real-binary regression uses default configuration with propagation disabled: dagu stop allows a ten-second abort handler to finish, then executes the exit handler and persists both as succeeded.
  • conformance/cli/TestStopPreservesHandlers uses static Unix and Windows fixtures for default cleanup and forced repeat-step cleanup. It checks that abort and exit handlers finish in order and the run persists Aborted. Both Unix cases fail against the original PR binary. Both cases pass on Linux and native Windows CI with the fix.
  • Supervisor tests cover server, scheduler, and start-all waiting for handlers beyond the step cleanup budget. Agent tests cover stops during init/terminal handlers, DAG timeout, concurrent stop requests, canceled stop callers, startup cancellation, and coordinator rejection.
  • Uncached race tests pass: go test -race -count=1 ./internal/runtime/... ./internal/launcher/... ./internal/cmd/....
  • make fmt and make check pass, including Windows/Darwin lint.
  • The native Windows process-tree regression passes in Windows CI, including parent/descendant termination and tracked run completion for SIGINT and SIGTERM.
  • Local make conformance passes with the new regression: 1,274 tests, 7 platform-specific skips.
  • GitHub CI, conformance, and both container jobs pass on 2d22ab39f: Linux/Windows unit and integration shards, compatibility, Linux/Windows/Darwin lint, and amd64/arm64 container smoke. E2E and Windows installer jobs were skipped by change detection.
  • Container smoke verifies step cleanup, an abort handler beyond the cleanup deadline, the exit marker, and persisted handler success through the shipped Tini entrypoint on amd64/arm64.