fix: keep SIGTERM handled during supervisor shutdown
Merged
YO
22d ago+62+62+01234567890123456789-10-10-01234567890123456789
Summary
A repeated SIGTERM killed server, scheduler, and start-all while propagated DAG runs were still cleaning up. In the Docker image this happens on every docker stop, so the signal-shutdown smoke test from #2885 is flaky.
Root cause
- After the first shutdown signal, every SIGTERM subscriber in the supervisor calls
signal.Stop, so SIGTERM falls back to its default action (terminate). - The image runs
tini -g -- /entrypoint.sh, and the entrypoint runsexec sudo ... dagu start-all. Without a TTY, sudo runs dagu in its own process group.docker stopmakes tini signal that whole group, so dagu gets SIGTERM directly and sudo relays a second copy. - When the relayed copy arrives after the handlers have stopped, start-all dies immediately. Runner cleanup, abort and exit handlers, and status persistence are all lost.
Failed run: https://github.com/dagucloud/dagu/actions/runs/36808047912/job/110196700869. The container exited about 70ms after SIGTERM. Only the HTTP server logged the signal; start-all never logged its own shutdown.
Changes
- When a supervisor receives its first shutdown signal, the receiving handler installs a process-wide SIGTERM absorber before it unsubscribes. That handler is
notifyShutdownContextforserverandstart-all, and the scheduler's tick loop forscheduler. A repeated SIGTERM is then ignored until the process exits. Process managers still escalate with SIGKILL after their own timeout. - Startup is unchanged: a SIGTERM that arrives before the supervisor subscribes still takes its default action.
- A second SIGINT still force-exits, as before. That behavior is covered by
TestSecondSignalDuringRunCleanupand theSecondInterrupttests. notifyShutdownContextnow always records the received signal as the cancellation cause, so the propagation and non-propagation modes behave the same.- Added
TestRepeatedTerminateDuringRunCleanupforserver,schedulerandstart-all. It failed for all three before the fix withsignal: terminatedand passes after.
Checklist
- Code follows the project style guidelines
- Self-review of the code has been performed
- Tests have been added or updated as needed
- Documentation has been updated as needed (not applicable)
- Changes have been tested locally (
go test ./internal/cmd/ ./internal/service/scheduler/ ./internal/cmn/signalctx/,make conformanceforconformance/cli,make check, Windows build)