#35388 ci(dev publisher): verify takes an idle owned runner before it waits on the fleet

The DEV publisher's verify job does 1.5 minutes of work but queued a median of 7 minutes and up to 67 minutes before it started (run 41814294095: created 10:28Z, started 11:36Z). It waits on glaeda-std-xcode-26.6 behind long fleet jobs. At 12:28Z that label had 0 runners online and 22 jobs queued, the oldest for 2 hours. Meanwhile three glaeda-root-std-xcode-26.6 runners on another Mac sat idle, with no job queued on any of their labels.

Leo, 2026-11-04: "that dev build took WAY too long to get to me ... A 'dev build' often doesn't require the full-ass thing I feel like".

Verify now takes an owned runner that is free right now, when there is one.

• scripts/ci/e2e_runner_pool.py --free-now prints an owned std label (a pool from CI_OWNED_POOL_SLOTS on the lane's Xcode pin, or its root label) that has an online, idle runner whose labels have nothing queued in the janitor snapshot. Otherwise it prints nothing.
• The picker already counts a pool family's whole queue against every runner in it. That count reads an idle runner carrying none of the queued labels as taken, so the existing pick could not see these runners.
• The runner list uses the routing App token; the snapshot uses the job's own token.
• The runner job's pool step asks --free-now last and lets its answer replace verify's label. tip-dmg-build keeps its own pick. With nothing free now, or a failed read, the existing pick stands unchanged.
• The persistent-pool marker now counts 2 jobs only when tip-dmg-build shares verify's label.

The trust gate is unchanged. Verify's runs-on text, its checks (sha256, ditto, codesign --verify --deep --strict, the signed-app and sidecar checks) and publish's dependency on verified == 'true' are untouched. Verify still runs only on a std-class owned Mac with the lane's Xcode pin, never light (the 188 MB download rule).

Risk: placement of one secret-free job. The worst case is that the idle runner is taken in the seconds between the pick and the job queuing; verify then waits on that label, which the owned-pool rescue watches, as it does today.

Validation

• New test test_free_now_finds_an_idle_runner_no_queued_job_waits_for replays the incident shape. Its negative cases are a queued label on the idle runner, busy and offline runners, the light class, another Xcode pin, no runners read, no snapshot, owned pools off, and a failed read. It also checks the token order and the workflow wiring.
• Fleet run 41830823124 at 523f855e1cb008abb1c3d6bb46df9fd41ad0122d passed 15 test files:
  • test_ci_pr_runner_pool (257 tests)
  • test_cmux_next_dev_tip_feed
  • test_ci_signing_keychain_is_deleted_before_bundle_code
  • test_cmux_next_dev_daily_dispatch
  • test_ci_publisher_mac_jobs_move
  • test_ci_owned_pool_rescue (480 tests)
  • test_ci_no_github_hosted_macos
  • test_glaeda_route_app_budget
  • test_ci_fork_runner_routing
  • test_ci_runner_set
  • test_delivery_workflows_cancel
  • test_publish_app_token_workflow
  • test_ci_quota_hold_gate
  • test_classic_nightly_build_runner
  • test_ci_simple_pool_picker
• actionlint 1.7.12 on the workflow reports only the SC2129 style note that main already has on this step.

Observed / Not run

• Observed: the tests above, and the incident numbers from the fleet dashboard and the run logs.
• Not run: a live publisher run. The pool step checks out the default branch's scripts and sign runs only on a main dispatch, so the first real pick happens after merge. The runner job's log line free now: ... shows what it chose.

Changelog

none

Agent session: agent session

🤖 Generated with Claude Code

---

^{Open in [code]smith · Need help on this PR? Tag @codesmith-bot with what you need.}

---

Summary by cubic

The DEV publisher's verify job now takes an owned runner that is free at queue time instead of waiting on the fleet, which previously delayed it by a median of 7 minutes and up to 67. The pool step asks --free-now, which names an owned std label (the configured pool or its root label, on the lane's Xcode pin) whose runner is online, idle, and has no queued job on any label it carries. When nothing is free or the read fails, the existing pick stands unchanged.

Notes

• tip-dmg-build keeps its own pick; verify's checks and publish's dependency on verified == 'true' are untouched.
• The persistent-pool marker now counts 2 jobs only when tip-dmg-build shares verify's label.
• The worst case is the idle runner being taken in the seconds between the pick and the job queuing; the owned-pool rescue then watches that label as it does today.

Summary by CodeRabbit

• Chores
  • Automated verification now prefers an owned runner that is online, idle, and free of queued work. If none qualifies, it retains the previous runner selection.
  • macOS test-pool tracking now reflects whether verification and disk-image tasks use the same runner.
  • Added checks for runner availability, queued work, missing state information, and workflow selection behavior.