fix: keep dedicated workers counted, busy and out of zombie restarts
Summary
Autoscaling a group of dedicated workers misread it in three ways, all from one cause: the worker's main loop hands each job to a dedicated subprocess through a small channel and, when that channel was full, waited on send().await until a slot freed, a job duration away. That loop is the only thing that writes the worker's ping and its occupancy.
- Worker count too low. The autoscaler counts workers that pinged in the last 30s. A busy dedicated worker stopped pinging for about a job's duration, so most pods went uncounted (
count of 1 < min_workers of 5with 5 pods up). Belowmin_workersthe autoscaler returns before it reads the queue, so it never scaled out. - Occupancy always 0. Time spent in dedicated subprocesses was never added to occupancy, so scale-in always fired.
- Jobs ran twice. A job's own ping (
v2_job_runtime.ping) is set only when it is pulled. A dedicated job that took longer thanZOMBIE_JOB_TIMEOUT(60s) between pull and completion, waiting in the channel or running, was restarted by the zombie detector while still in the subprocess, and ran a second time. Jobs pulled into the channel were also marked running, so they did not count as waiting for the scale-out thresholds.
Changes
worker.rs: the main loop no longer pulls while any dedicated channel is full. It sleepssleep_queue()and loops, so it keeps pinging, and the jobs stay in the queue where the autoscaler counts them. The concurrency semaphore is alwaysSemaphore::new(1), so the subprocesses already run one job at a time and this costs no throughput.common.rs:DedicatedWorkerState, shared by a worker's main loop and its dedicated subprocesses (found by worker name), holds two things:- The busy union. Dedicated jobs and the jobs the main loop runs inline both hold a
BusyGuard, so overlapping work counts once. For a worker with dedicated subprocesses, this union replaces the main loop's own timers inOccupancyMetrics. - The in-flight set. These are jobs handed to a subprocess that it isn't done with yet. The main loop adds a job before the send. The subprocess takes a
HandedOverguard on receipt, and dropping it releases the job: at completion, when the subprocess loop exits with jobs still queued (for example a draining generation during an in-place reload), or on an error before the job is queued. A job released unfinished is recovered by the zombie detection, as before.
- The busy union. Dedicated jobs and the jobs the main loop runs inline both hold a
worker.rs: a task of its own pings the in-flight jobs every 5s, keeping only those still running on this worker. It runs apart from the main loop, which can spend longer than the zombie timeout on a job it runs inline (a restricted-token job, a flow-runner wait).- EE (
dedicated_worker_ee.rs; windmill-ee-private#997, merged, and #998):handle_dedicated_processtakes theHandedOverguard when it receives a job, and holds the busy guard from the semaphore permit until the job's result is read.
One behaviour change: a dedicated job that hangs is no longer "restarted" after 60s. That restart was what caused the double runs. It now stays running, like a hung job on a regular worker.
Not covered here: a job handed to a flow runner still blocks the main loop on done_rx.await until the step finishes, so the worker's own ping can still lapse during a long flow-runner step. The in-flight task does not cover those jobs, which have the same zombie exposure. It's a separate path and worth a follow-up.
Test plan
- Repro before the fix (EE build, one
dediworker, a bun dedicated script sleeping 40s, 8 jobs, DryRun autoscaling): worker ping age climbed to ~35s per job, the autoscaler loggedcount of 0 < min_workers of 1with the worker up, and 5 of 8 jobs were zombie-restarted and ran twice ("job already completed by another worker"). The batch took ~9 min. - Same repro after the fix: worker ping age ⤠6s throughout,
occupancy_rate_15s= 1.00 while busy and 0 once idle, unstarted jobs stayedrunning = falseand the autoscaler scaled out on them ("5 jobs waiting >= full scaleout threshold"). 0 zombie restarts, 0 double runs, batch took 320s (8 Ć 40s). - 3 jobs of 90s each, every one past the 60s zombie timeout even while executing: all completed once, 0 zombie restarts, 280s total.
- A plain bun job on the default group still runs.
-
cargo test -p windmill-worker --lib dedicated_worker_counts: an inline job overlapping a dedicated one counts once, and dropping the dedicated guard releases its in-flight job. - After the review fixes: 5 dedicated jobs of 40s took 200s, 0 zombie restarts, 0 double runs, job pings ⤠1s old, occupancy 1.00 while busy and 0.00 once idle.
- Not exercised: a restricted-token dedicated job run inline while other dedicated jobs are in flight (the reason the in-flight ping has its own task).
š¤ Generated with Claude Code
Summary by cubic
Fixes dedicated worker autoscaling, which misread busy workers as idle and double-ran long jobs.
When a dedicated worker's channel is full, the main loop no longer parks on send().await (which froze the worker's ping and occupancy for a job's duration); it leaves the job queued and keeps looping.
- In-flight dedicated jobs are now pinged by their own task, so jobs longer than 60s aren't restarted as zombies and run twice.
DedicatedOccupancyrecords time spent in dedicated subprocesses and adds it to the worker's occupancy for scale-in decisions.
One behavior change: a hung dedicated job no longer restarts after 60s; it stays running like a hung regular-worker job.
Not covered: jobs handed to a flow runner still block the main loop and keep their zombie exposure; worth a follow-up.