fix(worker): stop gracefully on SIGINT and SIGTERM

Merged
TH
12d ago
+221-11

Summary

dagu worker now stops gracefully on SIGINT and SIGTERM. Before this change the worker installed no signal handler. A SIGTERM from systemd or Docker ended the process with exit 143, and Worker.Stop never ran. Running DAG runs were killed by the parent exit watcher, with no handlers and no final status.

As the issue says, a signal aware context alone is not enough. Run contexts derive from the worker context, so canceling it would cancel steps and handlers. So Worker.Stop now drains running tasks before it cancels anything.

Changes

  • runWorker installs notifyShutdownContext(ctx.Context, syscall.SIGINT, syscall.SIGTERM) first thing, the same call runCoordinator makes. On a signal it logs Worker shutting down and calls w.Stop with the parent context. w.Start gets the parent context too.

  • Worker.Stop stops the pollers first, so no new task is claimed. It then sends SIGTERM to each running agent through Agent.Signal, the call dagu start makes on a signal. Each run applies its max_clean_up_time_sec window and runs its lifecycle handlers. Stop waits for the running tasks, or for its own context, and only then cancels the internal context.

  • Task contexts no longer derive from the poll context. Stop cancels them through the internal context after the drain. A task that registers after Stop has started gets the signal at once. remote_handler.go registers agentInstance.Signal for the task's run.

  • New test TestCancellation_WorkerStopDrainsRun in internal/intg/distr/lifecycle_test.go. A step traps TERM, sleeps 1 s and writes a marker, and an exit handler writes a second one. The test checks both markers and the exit handler status right after Stop returns.

  • New test TestWorkerSignalShutdown in internal/cmd/signal_unix_test.go, next to TestCoordinatorSignalShutdown.

Scope note: the embedded worker in internal/engine/worker.go wraps the same Worker, so its Stop drains running tasks too.

There is no fixed shutdown timeout. Each run's cleanup window bounds its cleanup, and a second SIGINT forces the exit. A repeated SIGTERM is absorbed, as in the coordinator. I can add a configurable bound if you want one.

Related Issues

Closes #2957. The coordinator had the same gap in #2956, fixed in #2972, and this reuses its handler.

Tests

All runs on macOS arm64 with Go 1.27.0.

  • TestWorkerSignalShutdown fails on main's worker.go with signal: terminated and signal: interrupt. It passes on the branch with -count=5. TestCoordinatorSignalShutdown also passes there in the same run, five times out of five.

  • TestCancellation_WorkerStopDrainsRun fails without the internal/service/worker change, because the trap marker is missing. With the change it passes with -count=3.

  • I ran go test ./internal/intg/distr/ -count=1 -timeout 30m on both. Main: ok in 759.9 s, 123 PASS. Branch: ok in 527.8 s, 124 PASS, where the extra one is the new drain test and no other test changed status. TestExecution_QueuedDispatch_ConsumesBulkItems took 483.0 s on main and 351.8 s on the branch.

  • go test -race ./internal/service/worker/ is ok. golangci-lint 2.14.0 with the repo config reports 0 issues on the three changed packages, on darwin and with GOOS=windows. gofmt reports nothing and go vet ./internal/engine/ passes.

Not run: the full make test, the race detector on the distr suite, any Linux or Windows runtime test, and the UI. The engine worker has no test of its own here. It is covered only through the shared Worker code.

Checklist

  • Code follows the project style guidelines
  • Self-review of the code has been performed
  • Tests have been added or updated as needed
  • Documentation has been updated as needed
  • Changes have been tested locally

Summary by cubic

dagu worker now stops gracefully on SIGINT and SIGTERM instead of being killed by the default signal action. Previously the worker installed no signal handler: a signal from systemd or Docker ended the process with exit 143, Worker.Stop never ran, and running DAG runs were killed with no cleanup and no final status.

  • Worker.Stop now drains running tasks first: it stops polling, sends SIGTERM to each running step, and waits for its cleanup window, lifecycle handlers, and final status report before canceling anything.
  • Task contexts no longer derive from the poll context, so graceful shutdown doesn't cancel steps and handlers early. A second SIGINT forces an immediate exit.
  • The embedded worker in internal/engine/worker.go shares the same Worker code, so it drains running tasks too.

Summary by CodeRabbit

  • Bug Fixes
    • Workers now handle termination signals by stopping new work and allowing running tasks to complete their shutdown cleanup before the worker exits.
    • Worker shutdown waits for running tasks to finish cleanup, subject to the caller’s timeout. If the timeout expires, remaining work is canceled.
    • Added coverage for shutdown behavior on Unix systems, including handling of termination signals and task cleanup.