#35522 ci(runner pools): fail over only to a member seen starting jobs

scripts/ci/runner_pools.py moved a pool to its next unmarked member whatever that member had shown. On 2026-11-04 at 13:43Z the stall rule marked blacksmith-6vcpu-macos-26 dead on a slow but alive queue (run 41836952227 waited 34 minutes and then ran there). macos_26 then moved to blacksmith-12vcpu-macos-26, which had started nothing, and run 41841056674 sat queued there for 50 minutes until it was cancelled.

What changes:

• A member after a pool's first becomes the live one only on positive evidence that it starts jobs within STARTED_TTL (one hour, the janitor's BLACKSMITH_DEAD_MINUTES). That evidence is any of: the probe saw it start, or a fresh, complete janitor snapshot shows a job running there, a label_started time, or a dead-label entry's live member (start_evidence()).
• The evidence is kept in RUNNER_POOLS._started so a fallback chosen on it survives a pass whose reads fail. It does not move _updated, which still dates the marks and choices (e2e_runner_pool.lingering_dead()).
• A pool already on a fallback stays there while the pass has no fresh evidence about it (unknown stays unknown, as with dead marks).
• If no member is both unmarked and shown live, the pool keeps its first member, as it already did when every member was marked.
• When a pool's first member is marked, each pass logs every fallback it skipped.

Readers of RUNNER_POOLS (fromJSON(...).<pool>, _dead, _updated) are unchanged. The value only gains _started once some member has shown a start.

Tests (tests/test_ci_runner_pools.py, 51 pass) cover:

• the 13:43Z pass through main(): 6 vCPU is marked dead and macos_26 stays on it, then moves to 12 vCPU once the janitor shows a start there
• a stale start record not counting
• the sticky fallback when nothing is known, and its return to the first member when a fresh read shows no start
• _started leaving _updated alone
• start_evidence() ignoring stale or truncated input and owned labels

The existing tests that flipped to a fallback with no evidence now hand in a start.

🤖 Generated with Claude Code

Agent session: agent session

---

^{Open in [code]smith · Need help on this PR? Tag @codesmith-bot with what you need.}

---

Summary by cubic

Fixes runner pool failover so a pool only switches to a fallback that has shown it starts jobs within the last hour. On 2026-11-04 the stall rule marked a slow but alive 6 vCPU macOS queue dead, and macos_26 moved to 12 vCPU, which had started nothing — a run sat queued 50 minutes.

A member after a pool's first becomes live only on positive evidence: the probe saw it start, or a fresh, complete janitor snapshot shows a job running there, a label_started time, or a dead-label entry's live member. The evidence is kept in RUNNER_POOLS._started so it survives passes whose reads fail; _updated is untouched. A pool already on a fallback stays there while a pass has no fresh evidence about it, and with no member both unmarked and shown live, the pool keeps its first. Docs and tests updated accordingly.

Summary by CodeRabbit

• Bug Fixes
  • CI runner pools now switch to a fallback only when recent evidence shows it has started jobs. Existing fallbacks are retained when a check has no fresh information, and recent job-start evidence can keep a fallback eligible through temporary gaps.
  • If no suitable fallback is available, the pool uses its first member.
• Documentation
  • Updated the runner pool guidance to explain fallback selection and the evidence used.