ci(cmux-next): re-land Linux DEV signing with the signer executable, and guard script modes
Re-lands #25442, which was reverted in #25464. Changes from the reviewed #25442 are listed under "What changed since #25442"; everything else is #25442 as reviewed.
What changed since #25442
- Why it was reverted: the first DEV run on a green source build (run 40320904962) failed in sign before signing anything, with
./scripts/ci/rcodesign-sign-bundle-dev.sh: Permission denied(exit 126). The script landed in #25390 with git mode 100644. Its fixture test runs it throughbash, so nothing caught the mode. Verify, publish and tip were all skipped, so nothing was published. - Fix:
scripts/ci/rcodesign-sign-bundle-dev.shis now 100755. - New guard:
tests/test_workflow_scripts_are_executable.py, in ci-guards'cigroup and thelinux-guardlane. It reads every workflow with PyYAML (already installed for that group) and fails when a step'srun:script runs a tracked script by path with a git mode other than 100755. Modes come fromgit ls-files -s. Self-tests cover each matching position, each exemption and each kind of working directory.- Run by path:
./x.shin a command position. That means after anyNAME=valueprefixes, at the start of a line, after a separator, after$(or!, or afterif,while,until,env,exec,timeortimeout N. Interpreter calls are exempt:bash ./x.sh,v="$(bash ./x.sh)",python3 x.py,source ./x.sh. - Working directory: each
./pathis resolved against the step'sworking-directory, else the job's, else the workflow'sdefaults.run.${{ github.workspace }}resolves to the root. A directory built from any other expression is counted and skipped; there are none on main. Acdinside the script is not followed. - On main: the guard passes over 419 directly run scripts in 169 workflows. #25442's signer was the only failure.
- Mutations: clearing the bit fails on the right step in each case:
scripts/ci/rcodesign-sign-bundle-dev.sh:cmux-next-dev-daily.ymlsign.scripts/cmux-debug-cli.sh:ios-streamed-validate.yml, three calls behindCMUX_TAG=cistream.web/scripts/run-tests.sh:ci-web.ymlweb-tests, which runs it as./scripts/run-tests.shunderdefaults.run.working-directory: web.
- Run by path:
Third and last PR that moves cmux NEXT DEV signing off a Mac. #25390 added the Linux signer and #25406 the Linux tools; this one switches cmux-next-dev-daily.yml to use them. Before the switch, the DEV feed stalled for 6 h behind a single macOS runner label (#25371 moved sign to the 6 vCPU Mac).
Change
- sign runs on
blacksmith-4vcpu-ubuntu-2404(forks:ubuntu-24.04), still in thereleaseenvironment, still with no concurrency group:dev_app_linux.pyunpacks the source app and checks its identity, then stamps the tip feed, its key and the build number. The Sparkle key is derived with only the variable's name on argv.- tip's own appcast step still passes the private key on argv to
scripts/derive_sparkle_public_key.swift. That line is unchanged from main, andnightly.yml,release.ymlandauto-resume-nightly-notarization.ymlcall the script the same way. Taking it off argv means changing that script, which is on.github/release-surface.txt, so it gets its own PR instead of riding this one.
- tip's own appcast step still passes the private key on argv to
install-rcodesign.shinstalls the pinned rcodesign, thenrcodesign-sign-bundle-dev.shsigns the app.dev_app_linux.py zippacks the signed app as the same artifact verify and publish read.- There is no keychain on Linux. The signer keeps key material in its own 0700 temp directory and unsets the certificate secrets once the PEM exists.
- The workflow never sets
CMUX_TIMESTAMP(which would drop the time stamp and the Developer ID check); a test asserts it.
- tip builds the tip DMG (hdiutil ULMO), signs it with a temporary keychain that is deleted afterwards, and submits it to notary on its Mac. Sign used to do this. It first runs
codesign --verify --deep --stricton the signed app. Its rollback check and its single-writer group are unchanged. - verify runs Apple's
codesign --verify --deep --stricton the Linux-signed app, thenverify_signed_dev_app.py(thecodesign -dchecks the Mac signer used to make), then the sidecar check.- It writes
verified=truelast, and has nocontinue-on-errorand no|| truearound its checks. - It runs on an owned glaeda mini when one is free now, else on
blacksmith-6vcpu-macos-26(see "Where verify runs").
- It writes
- publish and tip both need
[sign, verify]with!cancelled() && needs.sign.result == 'success' && needs.verify.result == 'success' && needs.verify.outputs.verified == 'true'. A failed, skipped or cancelled verify blocks both. - Tests:
test_cmux_next_dev_tip_feed.py: the gates, tip owning the DMG, and noCMUX_TIMESTAMP.test_ci_signing_keychain_is_deleted_before_bundle_code.py: rule 8 (the Linux PEM signer) and the narrowed rule 6 with its self-test.test_cmux_next_dev_daily_dispatch.py: verify is no longercontinue-on-error.- The owned-pool rescue and the queue janitor list this workflow.
Where verify runs
Leo's routing rule (2026-10-24): "Nightly-next build, sign and publish run on any machine dev builds can use, in the routing order (Linux, owned Macs, Blacksmith last), with no pinned host"
So verify, the Mac check of the Linux-signed app, is offered to an owned glaeda mini first, with blacksmith-6vcpu-macos-26 as the fallback:
- A small Linux
runnerjob runs after sign. It runsscripts/ci/e2e_runner_pool.pythe wayiroh-release-gate.ymldoes, and names an owned label only when a mini is free now. There are no queue rounds, and only main's own dispatch on its first attempt gets a pick.- The routing App key is used in this job, not in verify.
- On an owned pick it uploads the
macos-pool-persistent-*andowned-pool-watchmarkers.
- verify runs on
${{ github.repository_owner != 'manaflow-ai' && 'macos-26' || (github.run_attempt == 1 && needs.runner.outputs.label || 'blacksmith-6vcpu-macos-26') }}. - Fallback timing:
- No free mini: verify goes to Blacksmith right away.
- A picked mini that leaves verify queued:
ci-owned-pool-rescue.yml's sweeper adopts the run within about a minute of the marker (it lists markers every minute). Once verify has been queued pastCI_OWNED_POOL_RESCUE_SECONDS(90 s by default), the sweeper cancels the run and re-runs the failed and cancelled jobs. Attempt 2 always takes Blacksmith. Usually that comes to 3 to 4 minutes plus Blacksmith's own queue, but it is an estimate, not a bound: the cancel can take up toCANCEL_WAIT_SECONDS(20 min) to settle, and this code does not bound Blacksmith's queue. - A picked mini that holds verify in runner setup (glaeda's capacity wait): the sweeper moves it only after
SETUP_WAIT_SECONDS(15 min) in setup, then cancels and re-runs as above. In that case the publish can miss the 30-minute freshness target. This PR keeps the shared 15-minute setup budget and does not add a faster setup budget for this workflow. The first watched run shows which case happens. - Nothing that has started is cancelled. sign and the placement job are already done, and publish and tip wait for verify.
- A mini that refuses verify fails it within seconds; the run finishes and the sweeper re-runs the failed jobs on Blacksmith.
owned_pool_rescue.DISPATCH_WORKFLOW_PATHSandqueue_janitor.OWNED_DISPATCH_WORKFLOWSlist this workflow, so the rescue and the janitor treat it likeiroh-release-gate.yml.tests/test_ci_owned_pool_sources.pypasses with it.
Sealed resources
The Linux signer excludes Contents/Resources/bin/cmux-tui-ssh/** (signed on their own and re-pinned in their manifest) and the code-free SwiftPM resource bundles from nested signing; they stay in the bundle as sealed resources. Apple's codesign --verify --deep --strict in verify checks that seal, and verify gates publish and tip. The first watched run checks that verify reported no "a sealed resource is missing or invalid".
Risk
A persistent mini runs DEV app code. The diff sidecar's handshake comes from the cmux NEXT DEV build that main's workflow just signed. A compromised build could leave something behind on that mini.
Mitigations:
- The job holds no secret, no signing key and no publish token, and runs in no environment.
- It works in a fresh
$RUNNER_TEMPdirectory that analways()step removes. - Only main's own dispatch gets an owned pick.
- The same minis already run cmux-next.yml's app tests on feat-cmux-next code.
No real-app rehearsal before merge. Downloading the unsigned DEV artifact needs actions: read, which fast-checks commands don't get, and heavy local runs are not allowed.
Why the risk is bounded:
- publish and tip both need verify to succeed with
verified == 'true', and verify runs Apple'scodesign --verify --deep --stricton a Mac. A bad Linux signature therefore costs a missed publish, never a bad build. - The first dispatched run after merge is watched. The workflow is reverted if sign or verify is red, or if DEV has not published within 30 minutes of merge.
Ruling
The coordinator's ruling: "yes, the verify job may run on owned glaeda minis with the Blacksmith 6-vCPU fallback, within these limits: verify job only: no secrets, no signing key, no publish token in that job; a fresh per-job workspace, removed after the job; rule 6 is narrowed to allow exactly this one job's label set, not widened in general, and the CI test that enforces rule 6 is updated to match; the PR body quotes Leo's rule and names the risk."
Rule 6 as it now reads in tests/test_ci_signing_keychain_is_deleted_before_bundle_code.py:
- A job that holds signing, notarization or release-publishing material (or runs in a release environment) runs no bundle code at all, not only outside the keychain window: a process that bundle code leaves running survives into the later steps of the same job. Bundle code runs in jobs without secrets on literal ephemeral runner labels: a job without that material that runs bundle code on what a release job made (it needs one, directly or through others) runs on a literal label, alone or after the fork short-circuit. The one exception is OWNED_VERIFY_JOBS: exactly the job and runs-on named there, which may take an owned Mac free now, holding no secret, outside any environment, not continue-on-error, in a fresh work directory it removes in an always() step. The notarization scripts run in such jobs only with their launch smokes deferred (CMUX_DEFER_LAUNCH_SMOKES) or skipped. STEP_LEVEL_ONLY lists the jobs that still rely on rule 1 alone.
The test enforces this: check_verify_runners() covers the whole tree, and self_test_verify_runners() refuses each of these:
- another runs-on for the allowed job;
- the allowed runs-on in another workflow, or on another job;
- a secret in the allowed job;
- a missing
always()removal; continue-on-error.
Numbers
Before: the last 4 successful publishes (runs 40285252357, 40248339313, 40243744618, 40239171524):
| Job | Runner | Queue | Run |
|---|---|---|---|
| sign | Blacksmith Mac | 290-655 s | 142-181 s |
| verify | Blacksmith Mac, advisory | 135-2417 s | 42-47 s |
| tip | Blacksmith Mac | 181-3475 s | 37-300 s |
| publish | Linux, needs sign only | 1 s | 12-16 s |
After: Linux sign (seconds of queue, about 3-4 min run), then verify on an owned mini when one is free (about 1 min) or Blacksmith 6 vCPU, then publish. That is one fewer Blacksmith macOS job per publish, and Apple's verifier becomes a gate rather than advisory. The real after-numbers come from the first dispatched run, which I will post on this PR.
Tests
On the exact tree: tip feed, keychain/rule 6 (14 jobs), dispatcher, test_dev_app_linux, test_verify_signed_dev_app, test_rcodesign_sign_bundle_dev, production secrets, runner label policy (33), owned-pool rescue, owned-pool sources, the self-hosted guard and YAML parsing all pass.
Agent session: agent session
🤖 Generated with Claude Code
Summary by CodeRabbit
- Release Improvements
- Daily development builds are signed and verified before publication to the rolling prerelease and tip feeds.
- Verification checks the signed app archive, its bundle identity and entitlements, and accompanying diff data before a release can proceed.
- Tip-feed DMG images are signed on macOS and submitted for notarization. Publication requires successful verification, and the notarization process waits for submission status before continuing.