fix: keep SIGTERM handled during supervisor shutdown

Merged
YO
22d ago
+62-10

Summary

A repeated SIGTERM killed server, scheduler, and start-all while propagated DAG runs were still cleaning up. In the Docker image this happens on every docker stop, so the signal-shutdown smoke test from #2885 is flaky.

Root cause

  • After the first shutdown signal, every SIGTERM subscriber in the supervisor calls signal.Stop, so SIGTERM falls back to its default action (terminate).
  • The image runs tini -g -- /entrypoint.sh, and the entrypoint runs exec sudo ... dagu start-all. Without a TTY, sudo runs dagu in its own process group. docker stop makes tini signal that whole group, so dagu gets SIGTERM directly and sudo relays a second copy.
  • When the relayed copy arrives after the handlers have stopped, start-all dies immediately. Runner cleanup, abort and exit handlers, and status persistence are all lost.

Failed run: https://github.com/dagucloud/dagu/actions/runs/36808047912/job/110196700869. The container exited about 70ms after SIGTERM. Only the HTTP server logged the signal; start-all never logged its own shutdown.

Changes

  • When a supervisor receives its first shutdown signal, the receiving handler installs a process-wide SIGTERM absorber before it unsubscribes. That handler is notifyShutdownContext for server and start-all, and the scheduler's tick loop for scheduler. A repeated SIGTERM is then ignored until the process exits. Process managers still escalate with SIGKILL after their own timeout.
  • Startup is unchanged: a SIGTERM that arrives before the supervisor subscribes still takes its default action.
  • A second SIGINT still force-exits, as before. That behavior is covered by TestSecondSignalDuringRunCleanup and the SecondInterrupt tests.
  • notifyShutdownContext now always records the received signal as the cancellation cause, so the propagation and non-propagation modes behave the same.
  • Added TestRepeatedTerminateDuringRunCleanup for server, scheduler and start-all. It failed for all three before the fix with signal: terminated and passes after.

Checklist

  • Code follows the project style guidelines
  • Self-review of the code has been performed
  • Tests have been added or updated as needed
  • Documentation has been updated as needed (not applicable)
  • Changes have been tested locally (go test ./internal/cmd/ ./internal/service/scheduler/ ./internal/cmn/signalctx/, make conformance for conformance/cli, make check, Windows build)