PR #29886 - manaflow-ai/cmux

Leo, 2026-11-01, on the #29821 note ("The nightly DMG signer is back on its existing Blacksmith Linux route (16 or 32 vCPU; ubuntu-24.04 applies only to forks)"): "surely we measured the signign we dont need 32/16 just for signing". Then: "surely github has linux hosted we can just use".

What changed

nightly.yml: signer-build, signer-app, signer-app-notary, signer-dmg and pick-nightly-sign-runner take GitHub-hosted ubuntu-24.04 on attempt 1 and blacksmith-4vcpu-ubuntu-2404 on a re-run. They never take 16 or 32 vCPU, whatever LINUX_RUNNER or the overflow switch holds.
cmux-next-dev-daily.yml: pick-sign-runner (the DEV daily's sign pick) takes the same route. The DEV signer itself is Apple codesign on an owned Mac, then blacksmith-6vcpu-macos-26, which is already the smallest Mac size; it is unchanged.
tests/test_ci_self_hosted_guard.sh: the hosted-runner guard accepts this one runs-on line as an exact exception, with the measurement beside it. Every other hosted label still fails.
tests/test_nightly_trusted_routing.py: a new test pins the signer route (attempt 1 hosted, attempts 2 and 3 on 4 vCPU, forks hosted) for every LINUX_RUNNER value. The evaluator learns >.
tests/test_ci_nightly_off_blacksmith.py: the light-job check accepts the sign pick's new route.

Why this doesn't repeat the #29821 take-back

#29801 moved signer-dmg to owned Macs, with a macOS rcodesign build (signer-build-macos) and an exclusive Mac claim. #29821 took that back to the trusted Linux route. Here the signer stays the same Linux rcodesign ELF on an x86_64 Ubuntu 24.04 runner, and the secret jobs stay on trusted ephemeral runners (GitHub-hosted, or Blacksmith on a re-run). There's no Mac claim, no second signer build, and no secrets on owned or fleet runners. The GCP fleet Linux pool (vars.CI_CONTROL_RUNNER) is not used for the secret jobs, because putting release secrets on it would change runner trust policy.

Stall-aware fallback

A re-run (attempt 2 or later) takes Blacksmith Linux 4 vCPU.
The pool signer path is opt-in and has a twin. If any signer job fails, is cancelled or is skipped, including a hosted job GitHub drops unstarted, build-sign-notarize-nightly (the macOS signer) signs the build (tests/test_nightly_signer_jobs.py keeps that hand-off).
Gap, as on Blacksmith before: a hosted job that stays queued without being dropped holds the macOS signer until it resolves, because timeout-minutes doesn't count queue time. The workflow comment's "never holds the nightly" overstates this; a queue-timeout escape is a filed follow-up.
The sign pick is continue-on-error, and the signing jobs' runs-on falls back when it has no output, as before.

Performance

Measured on HQ build-fleet/data/summary/summary.json (runtime_by_job, week 2026-W44, Blacksmith Linux 32 vCPU unless LINUX_RUNNER named 16). Blacksmith minutes are runner-minutes; vCPU-minutes assume 32 vCPU.

metric	before	after	delta	harness
signer-app wall (p50 / p90)	240 s / 278 s on 32 vCPU	notarization-bound; expected about the same on hosted 4 vCPU	n/a	HQ summary W44, n=236
signer-dmg wall (p50 / p90)	199 s / 242 s	about the same (notary wait)	n/a	HQ summary W44, n=224
signer-build wall (p50 / p90)	7 s / 9 s (cache hit)	about the same on a hit; the first hosted run builds rcodesign once (about 75 s on 32 vCPU, longer on 4)	n/a	HQ summary W44, n=511
pick-nightly-sign-runner wall (p50)	11 s Blacksmith / 9 s hosted	9 s hosted	-2 s	HQ summary W44, n=255 Blacksmith, 74 hosted
Blacksmith runner-min/day, signer jobs + pick	about 275 (signer-app 138, signer-dmg 109, signer-build 15, pick 13)	0 on attempt 1; re-runs only, at 4 vCPU	about -275	HQ summary W44 (n × mean / 7)
Blacksmith vCPU-min/day, same jobs	about 8,800	re-runs only, at 4 vCPU	about -8,800	same

signer-app-notary and the DEV pick-sign-runner have no row of their own in the summary, so they're not counted above. Both are short API jobs.
CPU per job isn't recorded for Blacksmith runs in the summary. The wall times are dominated by notarization waits and artifact transfer, not CPU.

Verification

Observed (local, Python unittest and the bash guard, at this head):

tests/test_nightly_trusted_routing.py, tests/test_ci_nightly_off_blacksmith.py, tests/test_nightly_signer_jobs.py, tests/test_ci_fork_runner_routing.py, tests/test_runs_on_labels.py, tests/test_ci_signing_secrets_need_release_environment.py, tests/test_supplychain_security.py: OK.
tests/test_ci_self_hosted_guard.sh: check_no_github_hosted_runners passes. check_no_self_hosted_fleet_runners fails the same way on base (cmux-next.yml GUI lines).
Same failure on base, not touched here: tests/test_runner_label_policy.py (CI_CONTROL_RUNNER unreported), tests/test_ci_blacksmith_linux_failover.py (cmux-next-chromium-currency.yml), tests/test_ci_signing_keychain_is_deleted_before_bundle_code.py (legacy sign jobs, stable-next.yml).

Not run: a nightly on the new route. The next nightly-next run is the end-to-end check, and I'll watch it sign.

Follow-ups (from the review at b0e61bf7)

A queue-timeout escape for a hosted signer that stays queued, and soften the workflow comment.
Measure the first hosted signer-build cold rcodesign build against its 20-minute timeout.
Pin the guard exception to the named jobs, not any matching line in the two files.
Rulebook: the matching exception goes in leo-workspace machines/claude-rulebook.md (lane/fleet1-rulebook-hosted-signer).

Changelog

none

Agent session: agent session

🤖 Generated with Claude Code

^{Need help on this PR? Tag @codesmith-bot with what you need.}

Summary by cubic

Moves the nightly signer jobs and sign picks from Blacksmith Linux 16/32 vCPU to GitHub-hosted Ubuntu 24.04 on attempt 1, with Blacksmith 4 vCPU on re-runs. Signing is notarization-bound, not CPU-bound (W44 p50 on 32 vCPU: signer-app 240 s, signer-dmg 199 s, signer-build 7 s from cache), so the large runners were wasted.

The route covers signer-build, signer-app, signer-app-notary, signer-dmg, pick-nightly-sign-runner, and the DEV daily's pick-sign-runner. signer-build keeps the early fleet placement (#29864) ahead of the hosted route on its own attempt. Secrets stay on trusted ephemeral runners (GitHub-hosted or Blacksmith); rcodesign stays the Linux build, and the macOS signer keeps its existing fallback hand-off. Tests pin the route for every LINUX_RUNNER value, and the self-hosted guard accepts the signer runs-on lines as exact exceptions.

Caveat

Known gap carried over from Blacksmith: a hosted signer that stays queued without being dropped holds the macOS signer until it resolves, since timeout-minutes doesn't count queue time. A queue-timeout escape is a filed follow-up; the next nightly-next run is the end-to-end check.