ci(nightly): build-nightly-app takes the warm nightly-only builder when it is idle

Merged
TE
4d ago
+182-7

Nightly freshness, with manaflow-ai/cmuxterm-hq#6106. On the dedicated route, build-nightly-app goes to any of the 10 glaeda-nightly-only-m4pro runners. Warm state is per runner, so a build on a listener that hasn't built before is cold. Run 41340244486 on cmux7s-mac-mini-glaeda-nightly-only-3 took 452 s to check out (14 s warm) and 282 s for Swift packages (1 s warm), then ran a cold Release compile.

cmuxterm-hq#6106 adds glaeda-nightly-only-m4pro-builder to one nightly-only runner on each Mac. This PR:

  • nightly_build_runner.py: on the existing nightly-only shortcut (attempt 1, nightly-next push or dispatch), when vars.CI_NIGHTLY_ONLY_BUILDER_RUNNER is set, the picker reads the runners once. It sends build-nightly-app (owned and runner) to the builder label only if a runner carrying it is online and idle right then. Helpers and light jobs stay on the plain label. In every other case the build takes the plain 10-runner label as today: builder busy, offline, unread (no token or a 403), read failed, or variable mistyped.
  • nightly.yml: pick-nightly-build-runner mints its existing routing token on that path only while the variable is set (the same App and permissions), and passes NIGHTLY_ONLY_BUILDER_LABEL. No runs-on changes.
  • runner_label_policy.py accepts the variable only as empty or exactly the builder label. ci-repo-variables.yml and ci-health-report.yml list it.
  • warm-sources.json: build-nightly-app lists deriveddata-reuse.

No pin: the builder label is chosen only when an idle runner carries it, and a new test in test_ci_runs_on_fallback.py fails if the label or its variable ever appears in a runs-on. The remaining race: two runs picking within seconds of each other, or a light job taking the builder between pick and queue, can make the build wait on that one runner. There is one builder per Mac, two in total.

Rollout: nothing changes until cmuxterm-hq#6106 is applied (fleet runner relabel), the label shows on both runners, and CI_NIGHTLY_ONLY_BUILDER_RUNNER=glaeda-nightly-only-m4pro-builder is set. Clear the variable to switch it off.

Tests: test_nightly_build_runner (54), test_nightly_dedicated_mini (9), test_ci_runs_on_fallback (24), test_ci_warm_sources, test_nightly_workflow_size, test_nightly_step_coverage, test_nightly_trusted_routing, test_runner_routing_table, test_ci_nightly_no_hosted_mac, test_nightly_admission_receipt and test_runner_label_policy pass. test_ci_linux_single_size and test_ci_self_hosted_guard.sh fail exactly as on base.

Follow-up for keeping caches per Mac instead of per runner: #31261.

Agent session: agent session

šŸ¤– Generated with Claude Code


View with [code]smith Autofix with [code]smith
Need help on this PR? Tag @codesmith-bot with what you need.


Summary by cubic

Sends build-nightly-app to a warm runner on the nightly-only reservation when one is idle, avoiding cold checkouts that previously added minutes per build.

  • When CI_NIGHTLY_ONLY_BUILDER_RUNNER is set, the picker reads the runners once and routes the app to the builder label only if a runner carrying it is online and idle right then; the read is bounded at 30 s so a slow listing can't drop the owned label.
  • If the builder is busy, offline, unread, the read times out, or the variable is unset or mistyped, the build takes the plain 10-runner label as today; the job never queues on the two-runner builder label.
  • Helper and light jobs keep the plain label; the routing token is minted on this path only while the variable is set.
  • The new variable is validated by runner_label_policy.py and listed in the repo-variable and health reports.

Rollout

  • Nothing changes until the runner label exists (cmuxterm-hq#6106) and CI_NIGHTLY_ONLY_BUILDER_RUNNER=glaeda-nightly-only-m4pro-builder is set. Clear the variable to switch it off.