web: auto-sizing learns from measured runs (per-VM peaks, tagged runs)

TE
8d ago
+340-27

Follows #26617 (landed), which added the picker.

Leo: "You know how we did that automatic routing, automatic sizing, automatic, you know, picking the best machine type logic for stuff? We want to have that for the cloud machines too, and just getting that cloud stuff just going, just simple, easy. Like ChatGPT work."

The spec says a pick learns from "past measured runs of the same repo and task kind". #26617 picks from the fleet's grants with a stub for measured runs. This PR records what each auto-sized machine actually used and feeds it to the next pick.

What this does

  • The run's peaks.
    • The guest reporter already posts CPU, memory and disk every 30 s. Until now, recordResourceUsage kept only the latest sample.
    • In the same atomic update, it now also keeps cmuxResourcePeak: the maximum memory and disk in use, plus the CPU sum and count for a mean. Peaks are per provider VM, so a new provider VM starts over.
    • Only an accepted sample counts. The 15 s coalescing is unchanged.
  • The run's tag.
    • autosize.repo.key (for example github.com/owner/name) is a new optional field.
    • When it is set, and the machine boots at the picked shape, the owner's VM (matched on user, provider and provider VM id) is tagged after the answer (runAfterResponse), off the create's critical path. The tag cmuxAutosize holds the repo, the task kind, the pick and the shape.
  • The next pick reads the runs.
    • A create with a repo key reads the owner's newest 20 tagged runs of that repo and task kind that recorded peaks and have ended (paused or destroyed). A running machine's peaks so far may be its idle start, so they don't count yet.
    • The read is bounded to 1.5 s. If it fails or times out, the pick falls back to the fleet table, so it never fails the create.
    • Each run becomes a MeasuredRun. The picker then applies the fleet's rule: p95 of peaks plus 20%. The reason line becomes, for example, "8 vCPU · 16 GB, picked from 3 earlier runs".
  • A run that filled its shape picks bigger next time. No OOM signal reaches the control plane yet, so a run whose peak reached 95% of its memory or disk counts as having run short on that axis (ranShort). The picker treats that as a floor that the other runs' percentile can't outvote.

Resize-up when short (design, not built here)

The Freestyle path already supports a grow-only resize (POST /api/vm/:id/resize, resizeVm). The next step:

  1. At a turn boundary, the hand-off checks the machine's fresh sample.
  2. If memory or disk is above 90% of its shape, it asks for the next shape on the ladder (images/sizes.ts) through that resize. The plan cap and pool reservation still apply.
  3. The recorded peak already makes the following pick start bigger.

GCP has no resize and keeps its shape.

Risk

Low to medium.

  • The new write path touches recordResourceUsage. It is the same row, in the same single UPDATE with the same coalescing predicate. It only adds a second top-level metadata key, built in SQL from the stored peak and the sample.
  • The new read is one indexed-by-owner query, only when a caller sends autosize.repo.key, and no client does yet.

Checks:

  • bun test: vm-autosize, vm-image-sizes, vm-create-option-policy, vm-route-auth.
  • The DB behavior tests ran against Postgres 16 with every migration applied: the new vm-autosize-runs-db (peaks across accepted and coalesced samples, the tag, and the read back into a pick) and the existing vm-resource-usage-db.
  • tsgo --noEmit, eslint and the complexity gate.
  • All of it ran on a Blacksmith Linux fast-check run.

🤖 Generated with Claude Code

Agent session: agent session