ci: size Blacksmith Linux jobs from one measured table; 88 jobs off 32 vCPU

TE
5d ago
+546-145

Leo, on 1,531 jobs queued on Blacksmith 32 vCPU: "ok. we clearly are not able to use them. so we should have resized a long-ass time ago."
And: "(im pretty sure blacksmith trimmed us off a bit just cuz we were allegedly costing like $10k a day) so we need to figure out solulus".

Blacksmith now limits how many vCPUs we can run at once, and that limit is small. A 32 vCPU job takes up the same room as eight 4 vCPU jobs. On 2026-10-31, the 32 vCPU size billed about 193,000 minutes ($12.4k). On 10-30 it billed 224,000 minutes ($14.4k). Most of these jobs are short.

What changes

  • One measured table. .github/runners.json now has a linux_sizes section. It lists every job that named blacksmith-32vcpu-ubuntu-2404, keyed <workflow>#<job>. Each entry gives the job's size and its failover. It also gives the measured [runs/day, mean min, p90 min] for each Blacksmith size the job ran on. Those numbers come from HQ build-fleet speed data for 10-30 and 10-31. Each entry ends with a one-line why. Prices are from blacksmith runners catalog, as recorded in HQ build-fleet/swift-linux-builder/blacksmith/RESULTS.md.
  • 88 jobs in 46 workflows drop off 32 vCPU. Each moves to the smallest size its measurement fits:
    • 4 vCPU, failover 8 (41 jobs): setup-bound jobs. They run under 2 minutes, or their 4 vCPU runs were within 1.5x of 32. Examples: the cmux-tui tree and publish bookkeeping, sentry-dsyms, pickers, SDK packaging, api-compat, web-frontend and go-pane.
    • 8 vCPU, failover 4 (44 jobs): cargo, swift, bun and playwright compiles and tests. On the minis these job classes use at most 6.2 cores at p90 (HQ mini_job_cores_by_class, 10-25..31). This group includes:
      • the CI guard group (3.3 min mean on 32; it ran 175-359 s even on 4 vCPU)
      • fast tier (cargo)
      • clippy
      • daemon tests
      • conformance
      • Swift on Linux
      • gallery advisory
    • 16 vCPU, failover 8 (3 jobs):
      • cmux-next checks: 6.2 min mean on 16 against its timeout. Its timeout goes from 10 to 15 minutes for the 8 vCPU failover.
      • the cmux-tui binary build legs: on 8 vCPU, macOS cross legs took 20-46 min.
      • cmux-tui.yml test: 9.3 min mean on 32.
  • Failover now goes to a smaller size. Before, the overflow failover sent 4 and 8 vCPU jobs to 16, and 16 to 32. Under a vCPU cap the size most likely to start is a smaller one. CI_CLOUD_OVERFLOW_SAVED has been set since 04:23Z, so every OVERFLOW && '32' || '16' job runs on 32 right now. tests/test_ci_blacksmith_linux_failover.py now also accepts 4↔8 and 16→8.
  • Fleet first is unchanged. Every route that put portable placement, a non-Blacksmith LINUX_RUNNER or LINUX_RUNNER_DARWIN_CROSS first keeps that order. Only the Blacksmith overflow member shrinks. No nightly job gains Blacksmith. The nightly-next-launch-smoke.yml Blacksmith literals only shrink (16/32 → 4/8).
  • Fork pull requests that took 32 vCPU now take 4 (cmux-next.yml web-bundles) or 8 (cmux-vm-clients.yml rust).

The test

tests/test_ci_blacksmith_linux_sizes.py is new and registered as linux-guard. It fails when:

  • a job's runner text can reach 32 vCPU without a linux_sizes entry, whether in runs-on, a matrix runner, an env or with value, or the linux_32 capability;
  • an entry holds 32 without why_32 or a named pending PR;
  • an entry's size or failover is not one of the labels its workflow names;
  • an entry names a job that no longer exists;
  • the pending count (PENDING = 51) grows or shrinks. It can only go down.

The tests that pinned 32 vCPU for a job now read its size from the table through tests/blacksmith_linux_sizes.py: linux compile runners, the artifacts runner set, the daemon tests gate, the guard group, machine-join, headless debug, the web bundle cache and the cmux-next checks workflow.

Only one job still justifies 32: cmux-tui-testbox-warmup.yml (the Blacksmith Testbox, dispatch-only, which records its label in its identity script).

Overlaps (left pending, not edited here)

fileopen PRjobs pending
ci-web.yml#30277, #29981 (and the carve-out for web-subarea-scope)7
fast-checks.yml#29934 / #299321
cmux-next-fast-gates.yml#301752
gallery-health.yml, web-validation.yml#298443
nightly.yml#30098, #30116, #30113, #30278, #3028338

Each pending entry has a target size. A follow-up PR moves them once those PRs land.

The largest share of 32 vCPU minutes does not come from literals. It comes from jobs that read vars.LINUX_RUNNER or fromJSON(vars.RUNNER_POOLS).linux. Main's pools.linux lists 32 first, and LINUX_RUNNER has been frozen on 32. That is fixed in main PRs: #30308 (mirror_writes) and the pool small-first PR from this lane. This PR does not touch them.

Performance

Before is measured: the 10-30/31 average of the moved jobs' runs/day × ceil(mean wall) at the size each ran. After is projected, for the same runs:

  • where the job has its own measured wall time at the new size, that time;
  • otherwise, the old wall time × √(old vCPU / new vCPU). This is conservative: setup-bound jobs do not slow down at all.

Prices: $0.002 per vCPU-minute (2 vCPU $0.004/min … 32 vCPU $0.064/min, blacksmith runners catalog).

metricbeforeafterdeltasource
Blacksmith 32 vCPU minutes/day (moved jobs)50,9880-50,988HQ build-fleet speed 2026-10-30/31; linux_sizes.measured
Blacksmith 16 vCPU minutes/day (moved jobs)25,52419,830-5,694same, projected
Blacksmith 8 vCPU minutes/day (moved jobs)077,700+77,700projected
Blacksmith 4 vCPU minutes/day (moved jobs)015,547+15,547projected
Blacksmith vCPU-minutes/day (room under the cap)2,039,9921,001,064-51%projected
Blacksmith $/day (moved jobs)$4,080$2,002-$2,078 (-51%)minutes × catalog price, projected
Still on 32/16 here, pending (32 vCPU / 16 vCPU min/day)84,624 / 91,265 ($5,416 / $2,920)follow-up—same data

Total Blacksmith spend was $21.3k on 10-30 and $19.3k on 10-31. Of that, 32 vCPU was $14.4k and $12.4k.

Compiles skipped or kept warm: nothing new is compiled. Every moved compile keeps its warm source from .github/warm-sources.json (actions-cache / rust-cache) unchanged. A smaller box reads the same cache.

Verification

  • Ran locally on the head:
    • the new test
    • test_ci_blacksmith_linux_failover: its one failure, cmux-next-chromium-currency.yml:42, is the same on base
    • test_ci_linux_compile_runners
    • test_cmux_tui_artifacts_linux_runner_set
    • test_cmux_tui_daemon_tests_gate
    • test_ci_tui_routes_avoid_nightly_label
    • test_cmux_tui_publisher_testing
    • test_feat_ci_guard_group
    • test_machine_join_e2e_workflow
    • test_cmux_tui_headless_debug_workflow
    • test_cmux_next_web_bundle_cache
    • test_ci_linux_single_size
    • test_ci_runner_set
    • test_runs_on_labels
    • test_ci_fork_runner_routing
    • test_ci_nightly_off_blacksmith
    • test_nightly_trusted_routing
  • I ran about 225 tests that mention runners or the changed workflows. The remaining red ones fail the same tests the same way on base fe2bcde8b54: test_cmux_next_checks_workflow, test_ci_change_areas, test_ci_warm_sources, test_car_guards, test_next_batch and others. test_ci_test_execution_registry fails here because its git fixture cannot push on a lane VM. tests/test_ci_self_hosted_guard.sh fails check_no_self_hosted_fleet_runners the same on base.
  • python3 scripts/cmux-next/prepush-gate.py: no new violations.
  • Not run: actionlint, which is not installed on this lane VM; CI runs it.

The repo CLAUDE.md says not to add CI tests (Lawrence, 2026-10-23). This test was asked for in Leo's resize order, which covers CI routing, so it is added here.

Changelog

none

Agent session: agent session
🤖 Generated with Claude Code