#14584 ci: E2E counts idle owned Macs live instead of from the snapshot

The org routing App's key went live at 07:38Z. Since then, pull request CI reads which owned runners are idle straight from the runners API. The E2E picker still estimated from the janitor snapshot plus every run created since it. So it kept reporting "every pool is full after replaying N newer run(s)" while 20 of 34 std runners sat idle (11:24Z).

Change

• test-e2e.yml: the runner job mints the same read-only App token as ci.yml (permission-administration plus permission-organization-self-hosted-runners, then a repository-only retry when the org grant is missing, as in #14678). It does so only for this repository's auto runs, while CI_PR_POOL_OWNED is 1 and the App is configured. Both steps use continue-on-error, so if no token can be minted the snapshot decides as before.
• e2e_runner_pool.py: with that token, the owned pools are counted from idle runners, and their capacity from online runners, the same way pull request CI does it (pr_runner_pool.live_owned_free, live_online and live_pools, #14713). One runners listing gives both.
  • Only pull request runs from the last LIVE_WINDOW_MINUTES are charged to an owned pool. An older run's jobs are already on runners, so they show as busy there. The rest of the snapshot window counts toward the Blacksmith pools only (ephemeral_since).
  • Every in-flight E2E run that names an owned pool is still charged to it. An E2E run can wait in its sibling job for up to 30 minutes before it holds a Mac, and the runners API can't see it then. This errs high by the few E2E runs already running.
  • Charge per E2E run on an owned pool. An E2E run is now charged the 1 machine it holds (owned_since). Before, it was charged as a replayed pull request run (placed, REPLAYED_RUN_JOBS). That over-charge affected the snapshot path too: 3 idle machines could not take a new E2E run while an older one was in flight.
  • Any error listing runners falls back to the snapshot.
• API cost: one more runs listing on the GITHUB_TOKEN, when reading live. The runners listing uses the App's own budget.
• tests/test_ci_self_hosted_guard.sh: allow-lists the two new continue-on-error mint steps.

Rebased onto main 2026-09-25: the queue-rounds Blacksmith fallback (rule()) now carries the live snapshot, capacity, ephemeral_since and owned_since too.

The iOS picker is getting the same change in #14542, from that PR's owner. Part of manaflow-ai/cmuxterm-hq#661.

Tests

• tests/test_run_e2e.py: 109 pass, 5 new:
  • live idle runners replace a stale full snapshot
  • only the live window's runs are charged
  • the token and owned-pool gating
  • live capacity is the online runners, not the slot count
  • the workflow's two mint steps and wiring
• Also passing: tests/test_ci_pr_runner_pool.py, tests/test_ci_self_hosted_guard.sh, scripts/ci/guards-local.sh --all (except 8 local-only failures in unrelated guards under macOS bash 3.2, which fail the same on main), and actionlint on test-e2e.yml.

🤖 Generated with Claude Code

Summary by CodeRabbit

• Improvements
  • E2E test runs now consider live owned-runner availability and recent workload when selecting a runner pool, helping direct jobs based on current capacity and demand.
  • When live runner information is unavailable, pool selection continues to use snapshot-based capacity data.
  • Runner-pool descriptions reflect the live or snapshot capacity used for the selection.