fix: keep dedicated workers counted, busy and out of zombie restarts

Merged
RU
2d ago
+206-2

Summary

Autoscaling a group of dedicated workers misread it in three ways, all from one cause: the worker's main loop hands each job to a dedicated subprocess through a small channel and, when that channel was full, waited on send().await until a slot freed, a job duration away. That loop is the only thing that writes the worker's ping and its occupancy.

  • Worker count too low. The autoscaler counts workers that pinged in the last 30s. A busy dedicated worker stopped pinging for about a job's duration, so most pods went uncounted (count of 1 < min_workers of 5 with 5 pods up). Below min_workers the autoscaler returns before it reads the queue, so it never scaled out.
  • Occupancy always 0. Time spent in dedicated subprocesses was never added to occupancy, so scale-in always fired.
  • Jobs ran twice. A job's own ping (v2_job_runtime.ping) is set only when it is pulled. A dedicated job that took longer than ZOMBIE_JOB_TIMEOUT (60s) between pull and completion, waiting in the channel or running, was restarted by the zombie detector while still in the subprocess, and ran a second time. Jobs pulled into the channel were also marked running, so they did not count as waiting for the scale-out thresholds.

Changes

  • worker.rs: the main loop no longer pulls while any dedicated channel is full. It sleeps sleep_queue() and loops, so it keeps pinging, and the jobs stay in the queue where the autoscaler counts them. The concurrency semaphore is always Semaphore::new(1), so the subprocesses already run one job at a time and this costs no throughput.
  • common.rs: DedicatedWorkerState, shared by a worker's main loop and its dedicated subprocesses (found by worker name), holds two things:
    • The busy union. Dedicated jobs and the jobs the main loop runs inline both hold a BusyGuard, so overlapping work counts once. For a worker with dedicated subprocesses, this union replaces the main loop's own timers in OccupancyMetrics.
    • The in-flight set. These are jobs handed to a subprocess that it isn't done with yet. The main loop adds a job before the send. The subprocess takes a HandedOver guard on receipt, and dropping it releases the job: at completion, when the subprocess loop exits with jobs still queued (for example a draining generation during an in-place reload), or on an error before the job is queued. A job released unfinished is recovered by the zombie detection, as before.
  • worker.rs: a task of its own pings the in-flight jobs every 5s, keeping only those still running on this worker. It runs apart from the main loop, which can spend longer than the zombie timeout on a job it runs inline (a restricted-token job, a flow-runner wait).
  • EE (dedicated_worker_ee.rs; windmill-ee-private#997, merged, and #998): handle_dedicated_process takes the HandedOver guard when it receives a job, and holds the busy guard from the semaphore permit until the job's result is read.

One behaviour change: a dedicated job that hangs is no longer "restarted" after 60s. That restart was what caused the double runs. It now stays running, like a hung job on a regular worker.

Not covered here: a job handed to a flow runner still blocks the main loop on done_rx.await until the step finishes, so the worker's own ping can still lapse during a long flow-runner step. The in-flight task does not cover those jobs, which have the same zombie exposure. It's a separate path and worth a follow-up.

Test plan

  • Repro before the fix (EE build, one dedi worker, a bun dedicated script sleeping 40s, 8 jobs, DryRun autoscaling): worker ping age climbed to ~35s per job, the autoscaler logged count of 0 < min_workers of 1 with the worker up, and 5 of 8 jobs were zombie-restarted and ran twice ("job already completed by another worker"). The batch took ~9 min.
  • Same repro after the fix: worker ping age ≤ 6s throughout, occupancy_rate_15s = 1.00 while busy and 0 once idle, unstarted jobs stayed running = false and the autoscaler scaled out on them ("5 jobs waiting >= full scaleout threshold"). 0 zombie restarts, 0 double runs, batch took 320s (8 Ɨ 40s).
  • 3 jobs of 90s each, every one past the 60s zombie timeout even while executing: all completed once, 0 zombie restarts, 280s total.
  • A plain bun job on the default group still runs.
  • cargo test -p windmill-worker --lib dedicated_worker_counts: an inline job overlapping a dedicated one counts once, and dropping the dedicated guard releases its in-flight job.
  • After the review fixes: 5 dedicated jobs of 40s took 200s, 0 zombie restarts, 0 double runs, job pings ≤ 1s old, occupancy 1.00 while busy and 0.00 once idle.
  • Not exercised: a restricted-token dedicated job run inline while other dedicated jobs are in flight (the reason the in-flight ping has its own task).

šŸ¤– Generated with Claude Code


Summary by cubic

Fixes dedicated worker autoscaling, which misread busy workers as idle and double-ran long jobs.

When a dedicated worker's channel is full, the main loop no longer parks on send().await (which froze the worker's ping and occupancy for a job's duration); it leaves the job queued and keeps looping.

  • In-flight dedicated jobs are now pinged by their own task, so jobs longer than 60s aren't restarted as zombies and run twice.
  • DedicatedOccupancy records time spent in dedicated subprocesses and adds it to the worker's occupancy for scale-in decisions.

One behavior change: a hung dedicated job no longer restarts after 60s; it stays running like a hung regular-worker job.

Not covered: jobs handed to a flow runner still block the main loop and keep their zombie exposure; worth a follow-up.