ci(nightly-next): dispatched and re-run builds take the owned-Mac pick too

Merged
TE
10d ago
+245-88

Why

Brief: "Today it routes only attempt-1 push runs to owned (glaeda) labels. workflow_dispatch runs and attempt-2+ re-runs never get the routing token. They fall through to Blacksmith macOS, which is stalled, so a dispatched or re-run tip nightly can never build on an owned mini. They must go through the same picker as attempt-1 push runs: Linux or owned Macs first, Blacksmith last."

What

  • Routing token: pick-nightly-build-runner now mints it for nightly-next push and workflow_dispatch runs at any attempt. Before, it minted only for a push on attempt 1.
  • Picker: scripts/ci/nightly_build_runner.py no longer refuses an owned pick when the attempt is above 1. It now outputs attempt, the attempt the pick was made for.
  • build-nightly-app runs-on: it takes owned only when needs.pick-nightly-build-runner.outputs.attempt == github.run_attempt, meaning the pick was made fresh for this attempt.
    • A push, a dispatch, or a full re-run picks again and gets an idle owned Mac first.
    • A re-run of failed jobs (how the rescue moves a refused or stuck job) keeps the earlier attempt's pick. So it takes blacksmith, and a refusing label is never taken again.
  • Watch marker: an owned pick outside attempt 1 of a push, which the rescue's workflow_run watch does not cover, uploads the owned-pool-watch marker so the sweeper adopts the run.
    • The job's owned output is set only if that upload succeeded. An owned job is never left without a rescue (Codex review finding, fixed). A Codex gpt-6-astra xhigh review returned SHIP on the rebased head.
  • Environment: the picker job runs in the existing ci-routing-agent environment, like the other dispatchable pickers (iroh-release-gate.yml, test-ios.yml). Now that a dispatch can mint the token, tests/test_supplychain_security.py needs this. That environment has no protection rules and no branch policy. No secret, permission or environment is added.

Unchanged: no evicting concurrency group, no cancel-in-progress, and only labels the rescue watches (pr_runner_pool.persistent, so glaeda-side-; glaeda-dev- is still never offered).

The main-side rescue change, manaflow-ai/cmux#26236, has landed (23bd7151). This PR is rebased onto #26048 (dedicated builder). dedicated_route()'s attempt-1-only guard is removed too, so a dispatch, or a re-run that picks again, tries the dedicated builder first, then an idle owned Mac, then Blacksmith. Main's rescue already watches the dedicated label.

Tests (cmux-ci test, Linux)

  • tests/test_nightly_build_runner.py: 27 tests OK. New tests:
    • a workflow_dispatch run and a run_attempt=2 run each get the routing token;
    • end to end, the picker's outputs feed into build-nightly-app's runs-on, and both runs land on the free glaeda-side-std-xcode-26.6. With a stale pick (a re-run of failed jobs), the same run lands on Blacksmith;
    • the marker condition and the owned output's gate on the upload.
  • test_nightly_trusted_routing.py, test_nightly_step_coverage.py, test_ci_nightly_off_blacksmith.py, test_ci_owned_pool_rescue.py, test_nightly_sign_runner.py: OK.
  • test_supplychain_security.py: this job no longer fails it. The remaining failure, pick-nightly-sign-runner, also fails on base.

Before: a dispatch, or attempt 2 with a free owned Mac, went to blacksmith-*-macos-26. After: both go to glaeda-side-std-xcode-26.6.

Known limit: the tick's cancelled-tip re-run (gh run rerun --failed) keeps the picker's earlier output when the picker had already succeeded. Those re-runs still take Blacksmith, which is the refusal-safe path.

Agent session: agent session

šŸ¤– Generated with Claude Code