feat: record and download a worker group's autoscaling decisions

Merged
RU
2d ago
+633-13

Summary

When a worker group does not scale the way someone expects ("workers stayed up for 42 hours", "it never scaled out"), there is nothing to look at after the fact. autoscaling_event only records actions taken. The ticks where the autoscaler decided not to act, and the inputs it decided on (worker count, occupancy, jobs waiting per tag), go only to the server log at info level.

This adds an on-demand recording of every autoscaling tick of a worker group, and an export from the Workers page that a customer can send us.

  • Opt-in and self-expiring. "Record decisions" records a group for 24h, then stops by itself, so it can't be forgotten on. Rows are kept 48h, so a recording can still be downloaded a day after it ends. A group writes one row per tick, about 4,300-5,800 rows (a few MB) per recorded day, and nothing when no recording is running.
  • Every tick, with its reason. A row holds the worker count, the 15s/5m/30m occupancy, the jobs waiting on each of the group's tags, and the decision. Decisions that change nothing carry their reason too, for example:
    • no change: within cooldown, last Scaleout 20s ago, cooldown 30s
    • no change: count 1 (min 1, max 3), occupancy 12.0/8.3/9.1% (scale in below 25%, scale out above 75%), no tag at 3 jobs waiting
    • no change: autoscaling is disabled or the worker group is paused
  • Download gives a JSON file with the recorded decisions, the group's autoscaling config, the server version and the recording window.

Changes

  • Migration: autoscaling_recording (group → until, started_by) and autoscaling_decision (one row per recorded tick), indexed on (worker_group, recorded_at).
  • EE autoscaling_ee.rs (windmill-ee-private companion PR):
    • apply_autoscaling now returns its decision as text on every path. Each former add_autoscaling_event(..).await?; return Ok(()) became return add_autoscaling_event(..).await, so the scaling actions are unchanged. The cooldown message now carries the numbers.
    • apply_all_autoscaling writes a row for each recorded group: the decision, an error: … when the tick failed, or why the group was skipped (disabled, paused, or invalid config).
    • Reading which groups record is diagnostics only: if it fails, it is logged and scaling goes on.
  • windmill-api-configs: two endpoints, both requiring the devops role.
    • GET /configs/autoscaling_diagnostics/{worker_group} returns the recording state, the decision count and the decisions, oldest first. include_decisions=false returns the state and count only.
    • POST /configs/autoscaling_recording/{worker_group} with {record: bool} starts (or extends) a 24h recording, or stops it. Both are audit-logged.
  • monitor.rs: the sweep deletes decisions older than 48h and expired recordings.
  • Frontend: AutoscalingDiagnostics.svelte sits in the Autoscaling events header (EE, devops/superadmin only). It shows "Record decisions", or "Recording until … · Stop", plus "Download". The count refreshes with the panel's refresh button, and every 30s while recording.
  • Telemetry (feature_usage): autoscaling_diagnostics / recording_started and downloaded, registered in FEATURE_USAGE_KINDS (EE). Plain counters, so no disclosure change.

Screenshots

autoscaling-recording

Test plan

  • EE build, a dedicated group dedi with DryRun autoscaling (min 1, max 3), recording started through the API. A row landed per tick, for example no change: within cooldown, last Scaleout 20s ago, cooldown 30s, no change: count 1 (min 1, max 3), occupancy -/-/-% …, and Full_Scaleout to 3: count of 0 < min_workers of 1 but queue >= full scaleout threshold of 3. This branch doesn't include #12540, so the rows also show that bug: a count of 0 while the worker was busy.
  • Recording the reports group, which has no autoscaling, writes no change: autoscaling is disabled or the worker group is paused.
  • UI (Playwright): Record decisions → "Recording until …" with Stop and Download. Download produced autoscaling-dedi-<time>.json, holding the autoscaling config, the version, the recording window and all decisions oldest first. Stop → back to "Record decisions", with Download still available. include_decisions=false returns the count only.
  • worker_config.autoscaling_recording_start and _stop appear in the audit log, and a feature_usage row for autoscaling_diagnostics / recording_started landed.
  • SQLX_OFFLINE=true cargo check --workspace --features quickjs,enterprise,private,license, plus the CE cargo check -p windmill-api-configs -p windmill-autoscaling. The 8 new .sqlx entries are exactly the new queries.
  • npm run check:fast: no errors in the changed files (the 4 that remain are on main).
  • Not exercised: a user without the devops role seeing the controls hidden. The gate is the same $superadmin || $devopsRole the Workers page uses for config actions.

🤖 Generated with Claude Code


Summary by cubic

Records every autoscaling decision of a worker group—including ticks where it decided not to scale, plus the inputs (worker count, occupancy, jobs waiting per tag) it based that on—so scaling behavior can be diagnosed after the fact. Previously those details only went to info-level logs, and autoscaling_event recorded actions taken, not the reasoning.

Details

  • Recording is opt-in per worker group, lasts 24h, and stops by itself; decisions are kept for 48h after the recording window.
  • Devops/admin users can start and stop a recording and download the full set of decisions as JSON from the Autoscaling events header on the Workers page; the controls show to those roles only.
  • Adds autoscaling_recording and autoscaling_decision tables (one row per tick, indexed on (worker_group, recorded_at)) and a 48h retention sweep in monitor.rs.
  • The EE autoscaler returns its decision as text on every path, so no-change ticks record their reason (cooldown, idle, disabled); reading recording state is best-effort and never blocks scaling.
  • Adds GET /configs/autoscaling_diagnostics/{worker_group} and POST /configs/autoscaling_recording/{worker_group} with {record: bool}, both devops-role gated and audit-logged.