ci: spread compile admission across minis (opt in), size owned pools from online runners

Merged
TE
2mo ago
+761-84

A std mini has two root runners, and a compile takes either free root. GitHub hands a root job to any idle root runner, so two compiles (8 to 10 of 14 cores each) can share one mini while another mini's root runners sit idle. Separately, CI_OWNED_POOL_SLOTS drifts from the fleet: root-std is set to 18 with 20 root runners online, std to 40 with 42.

Spread-first admission (opt in with CI_OWNED_SPREAD=1). A new job in ci-macos.yml, admission-placement, runs just before compile admission, and admission waits for it on attempt 1. It lists the runners through the route App (the same way late-placement does) and pins admission to an idle root runner on a mini none of whose root runners is busy.

  • It picks the mini first: a mini with a warm runner wins, otherwise the run ID hashes over the empty minis. It then takes a runner on that mini, so a mini with more root runners is not favored.
  • With no empty mini it falls back to an idle warm runner, then to the bare root label.
  • changes only passes the names of the warm runners through (admission_warm), so other runs' late placement gets no gap to take the pinned runner before admission queues.
  • The picker emits admission_runner and admission_warm on attempt 1 only.
  • A skipped or failed placement never blocks admission (!cancelled()); admission then falls back to pr_admission_runner.

Live capacity. When the runner listing is available, an owned pool's capacity is its online runners. CI_OWNED_POOL_SLOTS stays the fallback without the listing and still turns root routing on. When the variable over-counts (offline runners), the online count is lower, so this is the more conservative choice.

The prose about root runners now describes two root runners per std mini.

Tests, run locally:

  • Red: e92dca3b253 (tests only) gives 3 failures and 8 errors in tests.test_ci_pr_runner_pool and a missing script for tests.test_ci_admission_placement.
  • Green: 2181674a5ae passes picker, admission placement, late placement and rescue (300 tests), tests/test_ci_self_hosted_guard.sh, and tests/test_ci_change_areas.py (267 tests).
  • 6 other test_ci_* failures also fail on an untouched main checkout.

🤖 Generated with Claude Code

Summary by CodeRabbit

  • CI Improvements
    • Runner capacity now reflects live online runners when available. If live status cannot be retrieved, scheduling falls back to existing capacity information.
    • When spread-first placement is enabled, eligible admission runs can select an idle root runner on a mini without a busy root job, preferring warm runners and distributing selection across eligible minis.
    • If no such mini is available, placement falls back to an idle warm runner, then to the root-runner pool. Placement failures do not block admission.