fleet labels: a nightly slot carries only its own class's nightly label (held)
Leo-ask: ask-e9c646101f
What happened
From #1819 (and #1823 for already-registered runners) an xl nightly slot also carries glaeda-nightly-std-xcode-<v>. cmux's nightly-next signing jobs queue on that same label, so on 2026-11-04 eleven sign-nightly-identity-legacy jobs went to an M5 Ultra Studio's nightly slot, a host never enrolled for signing:
| start (UTC) | run | job |
|---|---|---|
| 18:12:24 | 41857891987 | 126762888312 |
| 18:12:57 | 41867447845 | 126765784354 |
| 18:13:20 | 41873130397 | 126766267412 |
| 18:13:34 | 41869470224 | 126766483859 |
| 18:16:55 | 41868557041 | 126768955485 |
| 18:18:36 | 41875126859 | 126769949111 |
| 18:23:38 | 41866781601 | 126772922067 |
| 18:28:46 | 41883617575 | 126775929518 |
| 18:30:13 | 41886296333 | 126776085402 |
| 18:40:57 | 41876590763 | 126782872915 |
| 19:01:07 | 41882065008 | 126794306476 |
All attempt 1 of a nightly-next push. The job's routing: vars.CI_NIGHTLY_SIGN_RUNNER is unset and pick-nightly-sign-runner gave no sign_owned, so runs-on fell to vars.CI_NIGHTLY_NEXT_BUILD_RUNNER = glaeda-nightly-std-xcode-26.6. GitHub delivered the job's secrets (signing, notarization and release-app entries, values masked) to the runner's Worker. glaeda-cmux-runner-hook refused each job in its pre-job hook (the signing hooks list at that commit did not list the hook), so no step used them: "Import signing cert" never started in any of the eleven, and only the always() keychain cleanup and "Complete runner" ran. Since 19:01:07Z no signing, notarization or nightly-label job reached an xl nightly slot. The one nightly job there was nightly-next-launch-smoke.yml's smoke at 19:04:57Z (run 41893380813), which holds only the GitHub token. The xl nightly-only builders took 18 build-nightly-app jobs on glaeda-nightly-only-xl-builder.
The change
Outcome: nightly signing stays on the trusted-only signer (glaeda-trusted-next-signing). Earlier heads of this PR added a signing-only label for std nightly slots (runner.nightlySigning). That commit is reverted here. A std nightly slot is a side instance of a PR mini whose other runners run pull request jobs as the same macOS user. State a PR job leaves behind (launch agents, shell rc, a tool on PATH, the shared git mirror, build caches) outlives the whole-Mac claim and could reach a later signing job's environment or keychain. Leo, 2026-11-05: "if we can make it safe. IF we can make it safe. the whole point was the blah blah belt and suspedners what if pr code reads the keys exfiltrates".
What is left:
- No std nightly label on xl slots.
nightly_classes()returns the slot's own class only, so an xl slot isglaeda-nightly-xl-xcode-<v>alone. The DEV freshness slot keeps its xl→std spill under a newdev_classes()(same body). Root and side spills are unchanged. - The reconcile removes the stale label.
widened_labels()drops a nightly slot label when the same runner carries that pool's nightly label at a wider class. That takes a staleglaeda-nightly-std-xcode-<v>off an xl slot registered between #1819 and this change. A std slot's own label, another Xcode's label and every other label stay, in order. This fixes router-review F1 at 1605c71.
With manaflow-ai/cmux#35918 the signing jobs stop asking for any shared label, so these two parts are defense in depth: the shared std nightly label no longer reaches an M5 Ultra either way.
Companion PRs, both held:
- manaflow-ai/cmuxterm-hq#7124 enrolls no slot. It adds a policy test that every signing host is trusted-only and carries no PR-shared label.
- manaflow-ai/cmux#35918 routes every secret-bearing nightly job to the signer label only.
Effect
-
Signing: unchanged in where it runs. It stays on the signer. After cmux#35918, no attempt falls back to a shared label.
-
Ultra builds: the dedicated xl builders and the Ultras' PR, root and DEV routes are untouched. The xl nightly slot stops taking overflow jobs queued on the std nightly label (#1819's reason). It keeps
glaeda-nightly-xl-xcode-<v>, which no signing job asks for. -
Rollout order, every step waiting on this HOLD:
- Merge this PR.
- Merge cmuxterm-hq#7124.
- Re-apply the xl nightly slot, or let the label reconcile set it to
widened_labels(), so the stale std label comes off. Check its registered labels. - Merge cmux#35918 last.
Nothing is relabelled here.
Tests
FleetLabelsModuleTest and ManifestLabelsTest (23) pass locally. So do the nightly-slot, DEV-slot and roots-take-side RunnerTest cases. The rewritten xl test asserts:
- the xl slot gets no std nightly label;
- the reconcile adds none, and removes a stale one idempotently;
- a std slot and another Xcode's label are kept;
- the DEV and root spills stay.
Agent session: agent session
🤖 Generated with Claude Code