ci(web): react-apps-check shards run on Blacksmith 16 vCPU (-1.36M vCPU-min/day), failing over to 32
The webview test shards (react-apps-check (1..4) and their base replay react-apps-base-check) move from Blacksmith 32 vCPU to Blacksmith 16 vCPU. The overflow failover still goes to the other size (now 32). No other job changes. (Fork PRs resolve to Blacksmith on this job, as they did before this PR; the review follow-up is filed separately.)
Leo, 2026-10-30:
i assume blacksmith prob cut us off cuz we were using like $10k a day so
y'know, we'll have to be more self-reliant and size the stuff better
and later:
i mean the linux things cant our minis just run them
This PR only resizes jobs. The shards install Playwright Chromium with apt (--with-deps), so they count as Linux-only and the mini fallback does not apply here. The table below marks which of the other top jobs could run on owned minis; that work belongs in follow-ups.
Why these jobs
They cost the most on Blacksmith. Over 2026-10-29/30 (the fleet's daily GitHub job export), the shards used about 99k wall minutes a day at 32 vCPU, roughly 3.2M vCPU-minutes. The same days also had 303 shard runs that the overflow failover had already sent to 16 vCPU, which gives a same-suite comparison:
| shard runs, 2026-10-28..30 | n | p50 | p90 | max |
|---|---|---|---|---|
| 32 vCPU, pull_request, success | 6830 | 972 s | 1377 s | 2251 s |
| 16 vCPU, pull_request, success | 257 | 1107 s | 1681 s | 2139 s |
| 32 vCPU, push, success | 1088 | 886 s | 1274 s | 1865 s |
| 16 vCPU, push, success | 46 | 1016 s | 1499 s | 1812 s |
On 16 vCPU the shards take 14% longer at p50 and cost half as much per minute. No 16 vCPU run hit the 40 min timeout or ran out of memory. The 4 vCPU OOM in run 37666132324 is why test_webviews_tests_run_on_a_runner_with_room_for_the_suite still forbids 4 vCPU.
Performance
| metric | before | after | delta | harness or run link |
|---|---|---|---|---|
| shard Blacksmith vCPU-minutes / day | ~3.17M (99k min Ć 32) | ~1.81M (113k min Ć 16) | -1.36M (-43%) | gh-jobs export 2026-10-29/30; the after value scales the measured 16 vCPU p50 ratio (1.14) |
| shard Blacksmith wall minutes / day | ~99k | ~113k | +14k | same |
| est. Blacksmith $ / day (list $0.004/min per 2 vCPU) | ~$6.3k | ~$3.6k | -$2.7k | same |
| shard p50 wall (PR) | 972 s | 1107 s | +135 s | 6830 vs 257 runs above |
| shard p90 wall (PR) | 1377 s | 1681 s | +304 s | same |
Top 10 cmux jobs by Blacksmith use (2026-10-29/30, per day)
| # | job (workflow) | wall min/day | vCPU-min/day | measured | runner today | smallest fit / owned option | Linux-only? |
|---|---|---|---|---|---|---|---|
| 1 | web / react-apps-check (1-4) (ci-web via ci.yml and cmux-next web) | ~99k | ~3.17M | p50 972 s PR, Test webviews step 765-898 s | 32 vCPU | 16 vCPU (this PR) | yes (Playwright apt deps) |
| 2 | gallery health / health (4 shards) | 44.3k | 716k | mean 619 s, render 448 s | 16 vCPU | 8 vCPU to measure; fewer shards on a PR that touches few entries | yes (Playwright apt deps) |
| 3 | cmux-tui artifacts / build aarch64-apple-darwin | 11.0k | 349k | mean 610 s | 32 vCPU | owned Mac builds it natively | no (darwin target; a mini builds it natively) |
| 4 | cmux-tui artifacts / build x86_64-apple-darwin | 10.7k | 339k | mean 594 s | 32 vCPU | owned Mac | no |
| 5 | cmux-next fast tier (cargo) | 9.9k | 282k | mean 217 s; cargo step 92 s at 32, 194 s at 16 | 32 vCPU | stays 32 (CPU-bound) or an owned Mac | no (cargo) |
| 6 | cmux-next checks | 16.4k | 264k | mean 346 s, mostly Python script tests | 16 vCPU | 4 vCPU or any owned mini | no |
| 7 | cmux-tui artifacts / build aarch64-unknown-linux-musl | 7.7k | 240k | mean 417 s | 32 vCPU | 16 vCPU | yes (musl) |
| 8 | cmux-tui artifacts / build x86_64-unknown-linux-musl | 7.4k | 231k | mean 403 s | 32 vCPU | 16 vCPU | yes (musl) |
| 9 | cmux-next guards / CI guard group | 12.4k | 221k | 145 s at 32, 277 s at 4 | 32 / 4 vCPU | 4 vCPU or any owned mini | no (Python guards) |
| 10 | Fast / checks (fast-checks.yml, dispatch on main) | 6.2k | 199k | mean 188 s | 32 vCPU | 16 vCPU | mixed |
Not in the vCPU top 10 but high in wall minutes:
cmux-tui artifacts / build-windows(20.2k on 4 vCPU Windows).nightly-guards / workflow-guard-tests / release-notary(18.3k on 4 vCPU; Python, so not Linux-only and could move to a mini).generated catch-up / catch up(9.5k on 2 vCPU, 279 s of it in checkout).
Risk
- Slower webview verdict. A PR's verdict comes about 2 min later at p50 and 5 min later at p90. The webview verdict gates landing (rulebook), so this is wall time on the landing path.
- Timeout tail. The slowest 16 vCPU run took 2139 s against the 2400 s
timeout-minutes: 40. The 1920 s retry deadline still stops retries before the uploads. If timeouts appear, the fix is a fifth shard, not going back to 32. - Memory. No 16 vCPU shard was SIGKILLed in 303 runs.
run-webviews-tests.shsizes its file and dev-server pools fromnproc, so it runs fewer at once on its own. - Failover. While
CI_CLOUD_OVERFLOW_SAVEDis set, the shards go to 32 vCPU, the switch's pairing for 16 (test_ci_blacksmith_linux_failover).
Tests
tests/test_web_validation.pynow expects 16 vCPU with the 32 failover. Run locally: 14 OK.tests/test_runs_on_labels.py: OK.tests/test_ci_blacksmith_linux_failover.py: 1 failure,app-platform.yml:45. It fails the same way on basececaf56b, so it is a base red and not from this change.pr-overlap: no open PR on feat-cmux-next overlaps this change. Prior art was read: #24927 set 32 vCPU when the base replay was split out, and #26439 added the failover. Neither had a cost reason for 32.
š¤ Generated with Claude Code
Agent session: agent session