feat: record and download a worker group's autoscaling decisions
Summary
When a worker group does not scale the way someone expects ("workers stayed up for 42 hours", "it never scaled out"), there is nothing to look at after the fact. autoscaling_event only records actions taken. The ticks where the autoscaler decided not to act, and the inputs it decided on (worker count, occupancy, jobs waiting per tag), go only to the server log at info level.
This adds an on-demand recording of every autoscaling tick of a worker group, and an export from the Workers page that a customer can send us.
- Opt-in and self-expiring. "Record decisions" records a group for 24h, then stops by itself, so it can't be forgotten on. Rows are kept 48h, so a recording can still be downloaded a day after it ends. A group writes one row per tick, about 4,300-5,800 rows (a few MB) per recorded day, and nothing when no recording is running.
- Every tick, with its reason. A row holds the worker count, the 15s/5m/30m occupancy, the jobs waiting on each of the group's tags, and the decision. Decisions that change nothing carry their reason too, for example:
no change: within cooldown, last Scaleout 20s ago, cooldown 30sno change: count 1 (min 1, max 3), occupancy 12.0/8.3/9.1% (scale in below 25%, scale out above 75%), no tag at 3 jobs waitingno change: autoscaling is disabled or the worker group is paused
- Download gives a JSON file with the recorded decisions, the group's autoscaling config, the server version and the recording window.
Changes
- Migration:
autoscaling_recording(group →until,started_by) andautoscaling_decision(one row per recorded tick), indexed on(worker_group, recorded_at). - EE
autoscaling_ee.rs(windmill-ee-private companion PR):apply_autoscalingnow returns its decision as text on every path. Each formeradd_autoscaling_event(..).await?; return Ok(())becamereturn add_autoscaling_event(..).await, so the scaling actions are unchanged. The cooldown message now carries the numbers.apply_all_autoscalingwrites a row for each recorded group: the decision, anerror: …when the tick failed, or why the group was skipped (disabled, paused, or invalid config).- Reading which groups record is diagnostics only: if it fails, it is logged and scaling goes on.
windmill-api-configs: two endpoints, both requiring the devops role.GET /configs/autoscaling_diagnostics/{worker_group}returns the recording state, the decision count and the decisions, oldest first.include_decisions=falsereturns the state and count only.POST /configs/autoscaling_recording/{worker_group}with{record: bool}starts (or extends) a 24h recording, or stops it. Both are audit-logged.
monitor.rs: the sweep deletes decisions older than 48h and expired recordings.- Frontend:
AutoscalingDiagnostics.sveltesits in the Autoscaling events header (EE, devops/superadmin only). It shows "Record decisions", or "Recording until … · Stop", plus "Download". The count refreshes with the panel's refresh button, and every 30s while recording. - Telemetry (
feature_usage):autoscaling_diagnostics/recording_startedanddownloaded, registered inFEATURE_USAGE_KINDS(EE). Plain counters, so no disclosure change.
Screenshots

Test plan
- EE build, a dedicated group
dediwith DryRun autoscaling (min 1, max 3), recording started through the API. A row landed per tick, for exampleno change: within cooldown, last Scaleout 20s ago, cooldown 30s,no change: count 1 (min 1, max 3), occupancy -/-/-% …, andFull_Scaleout to 3: count of 0 < min_workers of 1 but queue >= full scaleout threshold of 3. This branch doesn't include #12540, so the rows also show that bug: a count of 0 while the worker was busy. - Recording the
reportsgroup, which has no autoscaling, writesno change: autoscaling is disabled or the worker group is paused. - UI (Playwright): Record decisions → "Recording until …" with Stop and Download. Download produced
autoscaling-dedi-<time>.json, holding the autoscaling config, the version, the recording window and all decisions oldest first. Stop → back to "Record decisions", with Download still available.include_decisions=falsereturns the count only. -
worker_config.autoscaling_recording_startand_stopappear in the audit log, and afeature_usagerow forautoscaling_diagnostics/recording_startedlanded. -
SQLX_OFFLINE=true cargo check --workspace --features quickjs,enterprise,private,license, plus the CEcargo check -p windmill-api-configs -p windmill-autoscaling. The 8 new.sqlxentries are exactly the new queries. -
npm run check:fast: no errors in the changed files (the 4 that remain are onmain). - Not exercised: a user without the devops role seeing the controls hidden. The gate is the same
$superadmin || $devopsRolethe Workers page uses for config actions.
🤖 Generated with Claude Code
Summary by cubic
Records every autoscaling decision of a worker group—including ticks where it decided not to scale, plus the inputs (worker count, occupancy, jobs waiting per tag) it based that on—so scaling behavior can be diagnosed after the fact. Previously those details only went to info-level logs, and autoscaling_event recorded actions taken, not the reasoning.
Details
- Recording is opt-in per worker group, lasts 24h, and stops by itself; decisions are kept for 48h after the recording window.
- Devops/admin users can start and stop a recording and download the full set of decisions as JSON from the Autoscaling events header on the Workers page; the controls show to those roles only.
- Adds
autoscaling_recordingandautoscaling_decisiontables (one row per tick, indexed on(worker_group, recorded_at)) and a 48h retention sweep inmonitor.rs. - The EE autoscaler returns its decision as text on every path, so no-change ticks record their reason (cooldown, idle, disabled); reading recording state is best-effort and never blocks scaling.
- Adds
GET /configs/autoscaling_diagnostics/{worker_group}andPOST /configs/autoscaling_recording/{worker_group}with{record: bool}, both devops-role gated and audit-logged.