PR #28480 - manaflow-ai/cmux

Summary

DEV daily's verify and tip-dmg-build re-runs (cmux-next-dev-daily.yml, attempt 2 and later) take needs.runner.outputs.alternate. classic_nightly_build_runner.alternate() always returned the other Blacksmith macOS 26 size, so after the owned-pool rescue re-ran a run, those jobs queued on blacksmith-12vcpu-macos-26, a label that had started no job for about a day. The janitor's 18:37Z snapshot on 2026-10-30 showed:

label	queued	running	oldest queued	last start (`label_started`)
blacksmith-12vcpu-macos-26	180	0	1399 min	none
blacksmith-6vcpu-macos-26	77	5	552 min	18:39Z

The stall rule (blacksmith_stalled) looks only at queue age, so it calls both labels stalled and can't tell busy from dead. The snapshot already carries label_started (queue_janitor.label_started()), so this PR uses it:

simple_pool_picker.blacksmith_dead(): a label the snapshot lists with nothing running and no job start in the hour before the snapshot is dead. A snapshot up to six hours old still counts as evidence. A snapshot without label_started, or a label with a recent start or a running job, proves nothing.
classic_nightly_build_runner.blacksmith_route() never picks a dead label while another shows life. That holds even when the janitor has emptied the dead label's queue, so the stall rule sees nothing waiting there. With every label stalled it picks the label that still starts jobs.
alternate() is the other label only while the snapshot doesn't show it stalled (fresh snapshot) or dead (any recent snapshot). Otherwise it's the pick itself, the same rule nightly_sign_runner.alternate() already uses on feat-cmux-next.
main() keeps a snapshot that is too stale for the stall rule as evidence for dead labels.

If no label shows life, the old order holds. This replaces nothing in the routing order: the same two Blacksmith macOS 26 labels, after the owned picks.

Prior art: lawrencecchen's #27444 (and #27448 and #27451, stacked on it) solves the same DEV daily case with ONLY: 6vcpu, which pins both the pick and the alternate to 6 vCPU. This PR derives the choice from the snapshot instead, so it also stops routing to 6 vCPU if that label dies, and it uses 12 vCPU again once that label starts jobs. This PR is on main because the DEV daily runs main's copy of the script (it checks out the default branch).

Testing

python3 tests/test_classic_nightly_build_runner.py

base scripts (origin/main) with the new tests: FAILED (failures=6, errors=4); e.g. test_the_dev_daily_rerun_never_takes_the_dead_label gets alternate blacksmith-12vcpu-macos-26.
head: Ran 29 tests, OK.

Observed / Not run

Observed: that file, plus tests/test_ci_simple_pool_picker.py (46), test_ci_owned_pool_rescue.py (408), test_ci_pr_runner_pool.py (252), test_ci_runner_pools.py (27), test_seed_derived_data.py (66), test_nightly_next_cancelled_tip.py (12), test_ci_publisher_mac_jobs_move.py, test_cmux_next_dev_tip_feed.py and test_ci_change_areas.py, all passing locally on Linux.
Not run: a live DEV daily dispatch. The stuck runs keep their old runner outputs, so only new DEV daily runs pick up this change.

Changelog

none

Agent session: agent session

🤖 Generated with Claude Code

^{Need help on this PR? Tag @codesmith-bot with what you need.}

Summary by cubic

Stops DEV daily re-runs from queuing on a Blacksmith macOS 26 label that has stopped starting jobs.

The re-run alternate always returned the other Blacksmith macOS 26 label, so after an owned-pool rescue re-ran a run, verify and tip-dmg-build queued on blacksmith-12vcpu-macos-26 — 180 jobs, none running, nothing started for about a day — because the stall rule only looks at queue age. The fix derives the pick and alternate from the queue janitor's snapshot: a label with nothing running and no job start in the hour before the snapshot is dead, and it is skipped while another label shows life. A snapshot up to six hours old still counts as evidence; with no label showing life, the old order holds. A truncated snapshot proves nothing, since its listing can miss a running job or a start.

This replaces an alternative fix that pinned both labels to 6 vCPU. Deriving the choice from the snapshot instead also routes away from 6 vCPU if that label dies and returns to 12 vCPU once it starts jobs again.

Summary by CodeRabbit

Bug Fixes
- Nightly builds now avoid runner labels that appear inactive, including when the latest pool snapshot is unavailable or up to six hours old.
- When a runner is stalled or offline, builds select an eligible alternative; if no suitable large runner is available, they fall back to a non-dead option.
- Alternate runner output now reflects the label selected for the current pool state, including when the other candidate is stalled or inactive.
- Inconclusive evidence, such as snapshots showing no active labels, does not cause all runner labels to be treated as dead.