ci(nightly): signing jobs queue only on the signer's label (held)

Merged
TE
1d ago
+301-73

Leo-ask: ask-e9c646101f

What happened

nightly.yml's secret-bearing jobs fell back, after their owned pick, to vars.CI_NIGHTLY_NEXT_BUILD_RUNNER (glaeda-nightly-std-xcode-26.6) and then vars.CI_SIDE_LANE_RUNNER (glaeda-side-std-xcode-26.6). Nightly builds and PR side lanes share those labels, and glaeda also gave them to an M5 Ultra Studio, a host never enrolled for signing. On 2026-11-04 eleven sign-nightly-identity-legacy jobs (attempt 1 of a nightly-next push, CI_NIGHTLY_SIGN_RUNNER unset, no sign_owned pick) queued on glaeda-nightly-std-xcode-26.6 and ran on that Studio's nightly slot:

start (UTC)runjob
18:12:2441857891987126762888312
18:12:5741867447845126765784354
18:13:2041873130397126766267412
18:13:3441869470224126766483859
18:16:5541868557041126768955485
18:18:3641875126859126769949111
18:23:3841866781601126772922067
18:28:4641883617575126775929518
18:30:1341886296333126776085402
18:40:5741876590763126782872915
19:01:0741882065008126794306476

GitHub delivered each job's secrets (signing, notarization and release-app entries; values masked) to that runner. glaeda's pre-job hook refused every job before a step ran: "Import signing cert" never started.

The change

  • build-sign-notarize-nightly, sign-nightly-identity-legacy, generate-nightly-deltas, recover-nightly-next-notarization, publish-nightly, publish-nightly-next-github and publish-nightly-next-github-names keep the base branch's one-mini branch and their picker branch (sign_owned, sign_owned_pool). After that, they end on the literal 'glaeda-trusted-next-signing', the signer's label. The one-mini branch names that literal too, where it read vars.CI_NIGHTLY_SIGN_RUNNER before. The nightly-next push branch is gone: it came before the pick, so a push never used the pick's pin. Now the pick's pin comes first on every event (it names only the signer's runners), and the literal is the fallback when the pick found none. No repository variable chooses a signing job's label, so a stale or drifted variable cannot send signing secrets to a shared runner. They no longer fall back to CI_NIGHTLY_NEXT_BUILD_RUNNER, CI_SIDE_LANE_RUNNER or any shared pool literal.
  • promote-nightly-next keeps its base route. It runs only on a main dispatch, which the signer's hook refuses (it admits push and schedule only), so moving it to the signer would only break it. test_promote_keeps_its_earlier_route pins that route, and the shared-label test leaves it out with a comment saying why.
  • Both signing pickers (pick-nightly-sign-runner and find-nightly-next-recovery) offer only the signer's label, as a literal OWNED_LABELS. scripts/ci/signing_runner.py gains SIGNER_LABEL. nightly's owned pick (signer_only) keeps only runners that carry exactly that label, so a drifted OWNED_LABELS produces no pick instead of a shared runner. The picks in release.yml, the DEV daily and repair are unchanged.
  • scripts/ci/runner_label_policy.py reports CI_NIGHTLY_SIGNING_POOL or CI_NIGHTLY_SIGN_RUNNER as drifted unless it is glaeda-trusted-next-signing or unset. This is a drift report (run by ci-health-report.yml and ci-repo-variables.yml), not enforcement. The signing routes no longer read either variable; CI_NIGHTLY_SIGN_RUNNER is still read by promote-nightly-next's base route and other non-signing jobs.
  • The fleet-runner guard and the signing-keychain guard accept the route.
  • publish-nightly-next-github and publish-nightly-next-github-names only upload, but they hold the release App key. They get their own route: the signing pick's pool on its attempt, then the literal signer's label. Moving the upload off the signer's mini is not safe, so it stays on the signer. The upload's contention for that mini (run 41900945757) remains.
  • The secret-free verify jobs keep their routes.

Threat model

The question: could pull request code on the machine read the signing keys and exfiltrate them?

  1. Only trusted code runs on the signing host.
    • The host runs two kinds of runner:
      • Trusted pool runners, under glaeda's trusted_refusal (teamleaderleo/glaeda scripts/glaeda-cmux-runner-hook, lines 1441-1481 at a8ffe1ac). It admits merged-main push and schedule jobs, plus nightly.yml's signing jobs on the exact nightly-next push.
      • Dedicated nightly-only runners, under glaeda-nightly-only-admit. It admits non-signing nightly.yml jobs on main or nightly-next (workflow_dispatch included), and cmux-next.yml's release compile on a feat-cmux-next push.
    • Every admitted job runs merged code from a protected branch. Checkouts use github.sha, head_sha or build_sha (for example .github/workflows/nightly.yml:1213).
    • A pull request or fork job that names one of these labels can be handed to such a runner. The job-started hook refuses it before any step runs.
    • Routing no longer depends on any variable: every signing job's runs-on and both pickers' OWNED_LABELS name glaeda-trusted-next-signing as a literal, and the pick (signing_runner.py signer_only) emits only runners that carry exactly that label.
    • glaeda gives a trusted-only host no nightly slot (scripts/glaeda_fleet_labels.py:340) and no shared pool label.
    • In cmux, tests/test_nightly_signing_on_cmux15.py:122 (test_no_signing_job_ever_resolves_to_a_shared_label) pins that no signing job resolves to a nightly, side, root, std or nightly-only label, on any event or attempt, with or without one-mini mode, and with CI_NIGHTLY_SIGN_RUNNER and CI_NIGHTLY_SIGNING_POOL set to a shared label. It also pins that no signing route reads either variable.
    • tests/test_ci_signing_owned_mac.py:816 pins that the pick names only runners carrying exactly the signer's label, including when OWNED_LABELS drifts to a shared label.
    • Rejected alternative: signing on the std nightly slots. Other runners on those minis run pull request jobs as the same macOS user. State such a job leaves behind (launch agents, shell startup files, tools on PATH, shared caches) outlives a whole-Mac claim, and a same-user process can read the signing job's environment.
  2. The signing keychain lives only for the job.
    • scripts/ci/signing-keychain.sh:126 (create_keychain) deletes any earlier keychain first, then makes one with a random password that locks after 600 s idle (line 37).
    • delete_keychains (line 220) removes it. The if: always() "Cleanup keychain" steps run it (.github/workflows/nightly.yml:4245, :6196).
    • tests/test_ci_signing_keychain_is_deleted_before_bundle_code.py (check_job, line 293) requires bundle code to run only after the delete.
    • Residual risk. If the runner is killed mid-job, no step runs:
      • The keychain stays in ~/Library/Keychains, and its password file stays in the runner temp directory, until the next signing job's create_keychain or that runner's next job.
      • Every runner and service on the host runs as the same macOS user, so all of them can read both. All of them run only merged code (item 1).
      • While the keychain is unlocked, any same-user process can use it.
      • The protection is the exclusive claim: no other job starts, and running jobs drain first. The claim does not see a stray process outside _work, so this rests on the host running only trusted code.
  3. Signing secrets come only through the release-next environment.
    • Its deployment-branch policy is a GitHub environment setting, and no test pins it. (The comments at .github/workflows/nightly.yml:3456 and :5880 only describe it.)
    • tests/test_ci_production_secrets_protected_env.py:89 pins only that the jobs name the release-next environment. tests/test_ci_signing_secrets_need_release_environment.py pins that the signing secrets are read only in such jobs.
    • No pull request job is admitted on the signer (item 1).
  4. The exclusive publishers keep their whole-Mac hold, claim gate and post-job credential wipe, unchanged (tests/test_nightly_owned_minis_only.py:177 claim gate first, :196 credentials wiped after the job). The wipe, like the keychain delete, is a job step: it does not run if the runner is killed.

Rollout order (this PR merges last)

  1. teamleaderleo/glaeda#1841 merges. It stops giving xl nightly slots the std nightly label.
  2. The signer carries glaeda-trusted-next-signing (glaeda runner.trustedNextSigning, already set in cmuxterm-hq's manifest).
  3. Then this PR. manaflow-ai/cmuxterm-hq#7124 enrolls no host for signing.

Effect

  • Resume and re-run signing: a re-run, or an attempt whose pick is stale or empty, queues on the signer's label. Signing waits for the one signer rather than spilling to a shared mini.
  • Ultra nightly builds: untouched. Build jobs keep their routes, including the xl builders.
  • Not covered here: cmux-next-dev-daily.yml's "Sign cmux NEXT DEV" still picks on CI_SIDE_LANE_RUNNER.

Tests

The tests that pin these routes are updated: test_ci_signing_owned_mac, test_nightly_sign_runner, test_nightly_owned_minis_only, test_ci_nightly_off_blacksmith, test_nightly_trusted_routing, test_nightly_never_blacksmith, test_nightly_signing_on_cmux15, test_runner_label_policy, test_seed_derived_data, test_nightly_universal_build.sh, test_ci_self_hosted_guard.sh and the signing-keychain guard. The branch merges the base's one-mini mode, and test_nightly_one_mini passes.

test_no_signing_job_ever_resolves_to_a_shared_label is red/green:

  • on the base branch's nightly.yml it fails, with jobs resolving to glaeda-nightly-std-xcode-26.6;
  • on this PR's earlier head (8f12dcf, with the label since dropped) it fails, with glaeda-signing-xcode-26.6;
  • on 66b6fff, with both variables set to a shared label, it fails: the routes still read them, and a drifted value resolves to glaeda-nightly-std-xcode-26.6 or glaeda-side-std-xcode-26.6;
  • on this head it passes.

test_a_nightly_next_push_takes_the_pick_then_the_signer (a nightly-next push with a pick resolves to the pick's pin, and with none to the signer's label) fails at a179985, where the push branch came before the pick, and passes here.

Locally at a8a55c7, these pass: the routing, picker, policy, rescue, fallback, one-mini, promotion-admission and guard tests (the fleet-runner guard and the signing-keychain guard). The signing-keychain guard is red on the base branch (25 findings, from the base's one-mini vars.CI_NIGHTLY_MINI != '' && vars.CI_NIGHTLY_SIGN_RUNNER routes). Here it reports 11: the seven one-mini selectors, now vars.CI_NIGHTLY_MINI != '' && 'glaeda-trusted-next-signing', a form the guard does not parse, and promote-nightly-next's four base-route reads. Three other failures fail the same way on the base branch: two OwnedWorkdirSteps cases and test_seed_derived_data's test_the_trusted_pool_seeds_each_extra_root_as_its_own_lane.

Agent session: agent session

šŸ¤– Generated with Claude Code