feat(worker): bound the shutdown drain with worker.shutdown_timeout

Merged
TH
14d ago
+332-12

Summary

dagu worker now bounds its shutdown drain with worker.shutdown_timeout, default 60s. Before this change runWorker called Worker.Stop with the cobra context, which has no deadline. A run whose exit handler hangs kept the worker alive after SIGTERM until the supervisor sent SIGKILL, and the run stayed Running in the store.

When the bound expires, Stop cancels the remaining runs through the internal context, as it did before #3035. The canceled runs then get a fresh 10s to exit and the remaining cleanup a fresh 5s, so the canceled run reports Aborted before the process exits and Stop returns nil.

Changes

  • worker.shutdown_timeout (flag --worker.shutdown-timeout, env DAGU_WORKER_SHUTDOWN_TIMEOUT) sets the bound, and runWorker derives the Stop context from it through workerStopContext. The default of 60s sits under the 90s systemd TimeoutStopSec default, and 0 keeps the unbounded drain from #3035.

  • A negative value fails validateWorker. A value that doesn't parse logs a warning and falls back to 60s, so a typo can't remove the bound.

  • Once ctx is done, Worker.Stop gives the stopDone wait a fresh 10s context, then gives coordinatorCli.Cleanup, the health server stop and the OpenCode host close a fresh 5s context if that one has expired too. Before, all four got the expired context, so Stop returned before the canceled runs had exited.

  • Godoc for Worker.Stop in internal/service/worker/worker.go and in the root engine.go now states the drain contract, the deadline, the 10s exit window and the 5s cleanup window (#3048). The README env table, config.schema.json and the dagu worker help text list the key next to the systemd, docker stop and Kubernetes stop defaults, and ask for a supervisor stop timeout at least 15s above it.

Related Issues

Closes #3047 and closes #3048, the Worker.Stop godoc follow-up filed with it.

Tests

All runs on macOS arm64 with Go 1.27.0.

  • New TestWorkerStopDeadline/DeadlineCancelsRunAndWaitsForItToExit in internal/service/worker fails on main with "Stop should wait for the canceled task to exit". It passes on the branch with -count=3. The no-deadline subtest passes on both.

  • New TestCancellation_WorkerStopDeadlineCancelsHangingHandler in internal/intg/distr uses an exit handler that runs sleep 60 and a 4s Stop deadline. On main it fails with []ir.Status{3, 2} does not contain 1, because the run is still Running when Stop returns. On the branch it passes with -count=3 in 4.35 to 4.51s per run.

  • go test ./internal/intg/distr/ -count=1 -timeout 30m on the branch gives ok in 730.6s, 74 top level tests PASS and 0 FAIL.

  • New TestLoad_WorkerShutdownTimeout and TestWorkerShutdownTimeout cover the default, YAML, env, flag, zero, invalid and negative values on the branch.

  • go test -race gives ok for ./internal/service/worker/ (64.8s) and ./internal/cmd/ (304.7s), with 0 DATA RACE, and ./internal/cmn/config/ and ./internal/cmn/schema/ pass. golangci-lint v2.13.0 with the repo config reports 0 issues on the changed packages for GOOS darwin, linux and windows. gofmt -l and go fix -embedlit=false -diff are clean.

End to end, I ran dagu start-all and dagu worker from both binaries with a DAG whose exit handler runs sleep 120, and sent SIGTERM to the worker once the step ran.

WorkerResult after SIGTERM
main binarystill running after 32s, ended by SIGKILL, run left Running
--worker.shutdown-timeout=5sexit 0 after 5.35s, run Aborted, onExit failed
DAGU_WORKER_SHUTDOWN_TIMEOUT=3sexit 0 after 3.25s, run Aborted
no setting, default 60sexit 0 after 60.21s, run Aborted
--worker.shutdown-timeout=0still running after 32s, same as main

I didn't run the full make test, the race detector on the distr suite, the distr suite on main, or any Linux or Windows runtime test.

Checklist

  • Code follows the project style guidelines
  • Self-review of the code has been performed
  • Tests have been added or updated as needed
  • Documentation has been updated as needed
  • Changes have been tested locally

Summary by cubic

Binds the worker shutdown drain with worker.shutdown_timeout so a run whose exit handler hangs no longer keeps dagu worker alive after SIGTERM or gets stuck Running in the store. Closes #3047 and #3048.

  • runWorker previously passed the undeadlined cobra context to Worker.Stop; it now derives the Stop context from worker.shutdown_timeout (flag --worker.shutdown-timeout, env DAGU_WORKER_SHUTDOWN_TIMEOUT), which defaults to 60s and sits under systemd's 90s TimeoutStopSec default. Setting it to 0 restores the unbounded drain.
  • When the bound expires, Stop cancels the remaining runs and gives the shutdown steps a fresh 10s context, so canceled runs report their final status before the process exits.
  • A negative value fails validation; an unparseable value logs a warning and falls back to 60s so a typo can't remove the bound.
  • README, config schema, and dagu worker help document the key next to the systemd, Docker, and Kubernetes stop defaults, and the Worker.Stop godoc now states the drain contract.

Summary by CodeRabbit

  • New Features
    • Workers now support a configurable shutdown timeout, defaulting to 60 seconds. During shutdown, they stop polling and drain running DAG runs before canceling any that remain.
    • Set the timeout to 0 to wait without a time limit. Configure the supervisor’s stop timeout to be longer than the worker’s.
  • Documentation
    • Added guidance on worker shutdown behavior and timeout configuration.