ci(dev-daily): verify and tip-dmg-build pick only std owned Macs

Merged
TE
6d ago
+68-0

Summary

In the cmux NEXT DEV daily, verify and tip-dmg-build each download the 188 MB cmux-next-dev-daily-signed artifact. The runner job's owned pick usually sent them to the light owned pool, whose minis download about 5x slower than std minis on the same network. This PR limits that pick to the std class.

  • scripts/ci/e2e_runner_pool.py: new --owned-classes std,light,xl option. An auto pick names or waits on owned pools of those classes only, and an unknown class is refused (exit 2). It filters the same way --blacksmith-labels already does, after observe().
  • scripts/ci/simple_pool_picker.py: new owned_class(label) helper.
  • .github/workflows/cmux-next-dev-daily.yml: the runner job's pick() passes --owned-classes std. verify and tip-dmg-build now take one of three routes: a free std mini, Blacksmith room on their own fallback label, or a wait on std (which the owned-pool rescue still watches). Re-runs still go to Blacksmith as before.

Evidence (fleet dashboard job log, 24 h to 2026-10-30):

joblight poolstd pool
Verify signed cmux NEXT DEV85 of 90 jobs, median 895 s, 71 hit the 15 min limitmedian 41 s
Build the unsigned cmux NEXT DEV tip DMG80 of 84 jobs, median about 1050 s, 35 cancelledmedian 241 s
"Download the signed app" step736-974 s (2.2-2.8 MB/s from blob storage)about 15 s (11.5-12.7 MB/s, same gateway, 1000baseT)

Performance

metricbeforeafterdeltaharness or run link
verify runtime895 s median (light), 71 of 90 at the 15 min limit41 s median (std)about -850 s per rundashboard jobs.json?since=24h + batched GraphQL step timings; after: the next DEV daily run after merge
tip-dmg-build runtimeabout 1050 s median (light)241 s median (std)about -800 s per runsame
owned Mac time on the two jobsabout 45 h/dayabout 5 h/dayabout -40 Mac h/daysame

Incremental builds and the SSD cache: neither job compiles. Both download the signed app the sign job made, so no cold compile is added anywhere and the shared fleet compile cache is not touched. Its only effect is where those two download-bound jobs run.

Risk (low): when std is busy and Blacksmith's own label has no room, verify waits on std instead of starting at once on a light mini. On light it ran 15 minutes and usually timed out, so the wait still comes out ahead. The light pool stays available to every other caller (no default changes).

Related open PRs: #27444, #27448 and #27451 route these jobs' Blacksmith fallback away from the stalled 12 vCPU label. They touch the same workflow but not the owned pick, so they don't overlap with this one.

Testing

  • python3 -I tests/test_ci_pr_runner_pool.py: 254 pass. New tests cover these cases:
    • a free light pool is taken without --owned-classes;
    • with --owned-classes std, a busy std pool is waited on;
    • Blacksmith room on the jobs' own label is taken before the free light pool;
    • a free std pool is still picked first;
    • an unknown class is refused.
  • python3 -I tests/test_cmux_next_dev_tip_feed.py passes, with a new check that the pick passes --owned-classes std.
  • Also passing: test_ci_simple_pool_picker.py, test_ci_owned_pool_rescue.py, test_ci_fork_runner_routing.py, and actionlint on the workflow.

Observed / Not run

  • Observed: the test runs above, the 24 h job and step timings, and a download probe on the light and std minis.
  • Not run: a DEV daily dispatch with this change. The workflow dispatches from main, so the first run after merge is the after-run.

Changelog

none

Agent session: agent session

šŸ¤– Generated with Claude Code


View with [code]smith Autofix with [code]smith
Need help on this PR? Tag @codesmith-bot with what you need.


Summary by cubic

Restricts the cmux NEXT DEV daily verify and tip-dmg-build jobs to std-class owned Macs so the 188 MB signed-app download no longer lands on the light pool's minis, which took 736-974 s for it against about 15 s on std. Over 24 h to 2026-10-30, 85 of 90 verify jobs ran on light with an 895 s median (71 hit the 15 min limit) versus 41 s on std.

  • Adds --owned-classes to e2e_runner_pool.py; an auto pick names or waits on owned pools of only those classes, and an unknown class exits with code 2.
  • Adds an owned_class() helper to simple_pool_picker.py.
  • Passes --owned-classes std from the DEV daily runner job, so the two jobs take a free std mini, Blacksmith room on their own fallback label, or wait on std.
  • When std is busy and Blacksmith has no room, verify waits on std instead of starting on light; the wait wins since most light runs timed out. Other callers keep the default all-classes behavior.
  • Related PRs #27444, #27448, and #27451 touch the same workflow but not the owned pick, so they don't conflict.

Summary by CodeRabbit

  • Chores
    • The daily development workflow now selects only standard-class owned runners. Other owned runner classes are excluded from selection and waiting for that workflow.
    • The runner picker now supports filtering by owned runner class; without a filter, it continues to consider all classes.
  • Tests
    • Added coverage for class filtering and invalid class values.