ci(cmux-next): sign the DEV app on Blacksmith Linux, verify it with Apple's codesign on an owned mini first

Merged
TE
25d ago
+460-150

Third and last PR that moves cmux NEXT DEV signing off a Mac. #25390 added the Linux signer and #25406 the Linux tools; this one switches cmux-next-dev-daily.yml to use them. Before the switch, the DEV feed stalled for 6 h behind a single macOS runner label (#25371 moved sign to the 6 vCPU Mac).

Change

  • sign runs on blacksmith-4vcpu-ubuntu-2404 (forks: ubuntu-24.04), still in the release environment, still with no concurrency group:
    • dev_app_linux.py unpacks the source app and checks its identity, then stamps the tip feed, its key and the build number. The Sparkle key is derived with only the variable's name on argv.
      • tip's own appcast step still passes the private key on argv to scripts/derive_sparkle_public_key.swift. That line is unchanged from main, and nightly.yml, release.yml and auto-resume-nightly-notarization.yml call the script the same way. Taking it off argv means changing that script, which is on .github/release-surface.txt, so it gets its own PR instead of riding this one.
    • install-rcodesign.sh installs the pinned rcodesign, then rcodesign-sign-bundle-dev.sh signs the app.
    • dev_app_linux.py zip packs the signed app as the same artifact verify and publish read.
    • There is no keychain on Linux. The signer keeps key material in its own 0700 temp directory and unsets the certificate secrets once the PEM exists.
    • The workflow never sets CMUX_TIMESTAMP (which would drop the time stamp and the Developer ID check); a test asserts it.
  • tip builds the tip DMG (hdiutil ULMO), signs it with a temporary keychain that is deleted afterwards, and submits it to notary on its Mac. Sign used to do this. It first runs codesign --verify --deep --strict on the signed app. Its rollback check and its single-writer group are unchanged.
  • verify runs Apple's codesign --verify --deep --strict on the Linux-signed app, then verify_signed_dev_app.py (the codesign -d checks the Mac signer used to make), then the sidecar check.
    • It writes verified=true last, and has no continue-on-error and no || true around its checks.
    • It runs on an owned glaeda mini when one is free now, else on blacksmith-6vcpu-macos-26 (see "Where verify runs").
  • publish and tip both need [sign, verify] with !cancelled() && needs.sign.result == 'success' && needs.verify.result == 'success' && needs.verify.outputs.verified == 'true'. A failed, skipped or cancelled verify blocks both.
  • Tests:
    • test_cmux_next_dev_tip_feed.py: the gates, tip owning the DMG, and no CMUX_TIMESTAMP.
    • test_ci_signing_keychain_is_deleted_before_bundle_code.py: rule 8 (the Linux PEM signer) and the narrowed rule 6 with its self-test.
    • test_cmux_next_dev_daily_dispatch.py: verify is no longer continue-on-error.
    • The owned-pool rescue and the queue janitor list this workflow.

Where verify runs

Leo's routing rule (2026-10-24): "Nightly-next build, sign and publish run on any machine dev builds can use, in the routing order (Linux, owned Macs, Blacksmith last), with no pinned host"

So verify, the Mac check of the Linux-signed app, is offered to an owned glaeda mini first, with blacksmith-6vcpu-macos-26 as the fallback:

  • A small Linux runner job runs after sign. It runs scripts/ci/e2e_runner_pool.py the way iroh-release-gate.yml does, and names an owned label only when a mini is free now. There are no queue rounds, and only main's own dispatch on its first attempt gets a pick.
    • The routing App key is used in this job, not in verify.
    • On an owned pick it uploads the macos-pool-persistent-* and owned-pool-watch markers.
  • verify runs on ${{ github.repository_owner != 'manaflow-ai' && 'macos-26' || (github.run_attempt == 1 && needs.runner.outputs.label || 'blacksmith-6vcpu-macos-26') }}.
  • Fallback timing:
    • No free mini: verify goes to Blacksmith right away.
    • A picked mini that leaves verify queued: ci-owned-pool-rescue.yml's sweeper adopts the run within about a minute of the marker (it lists markers every minute). Once verify has been queued past CI_OWNED_POOL_RESCUE_SECONDS (90 s by default), the sweeper cancels the run and re-runs the failed and cancelled jobs. Attempt 2 always takes Blacksmith. Usually that comes to 3 to 4 minutes plus Blacksmith's own queue, but it is an estimate, not a bound: the cancel can take up to CANCEL_WAIT_SECONDS (20 min) to settle, and this code does not bound Blacksmith's queue.
    • A picked mini that holds verify in runner setup (glaeda's capacity wait): the sweeper moves it only after SETUP_WAIT_SECONDS (15 min) in setup, then cancels and re-runs as above. In that case the publish can miss the 30-minute freshness target. This PR keeps the shared 15-minute setup budget and does not add a faster setup budget for this workflow. The first watched run shows which case happens.
    • Nothing that has started is cancelled. sign and the placement job are already done, and publish and tip wait for verify.
    • A mini that refuses verify fails it within seconds; the run finishes and the sweeper re-runs the failed jobs on Blacksmith.
  • owned_pool_rescue.DISPATCH_WORKFLOW_PATHS and queue_janitor.OWNED_DISPATCH_WORKFLOWS list this workflow, so the rescue and the janitor treat it like iroh-release-gate.yml. tests/test_ci_owned_pool_sources.py passes with it.

Sealed resources

The Linux signer excludes Contents/Resources/bin/cmux-tui-ssh/** (signed on their own and re-pinned in their manifest) and the code-free SwiftPM resource bundles from nested signing; they stay in the bundle as sealed resources. Apple's codesign --verify --deep --strict in verify checks that seal, and verify gates publish and tip. The first watched run checks that verify reported no "a sealed resource is missing or invalid".

Risk

A persistent mini runs DEV app code. The diff sidecar's handshake comes from the cmux NEXT DEV build that main's workflow just signed. A compromised build could leave something behind on that mini.

Mitigations:

  • The job holds no secret, no signing key and no publish token, and runs in no environment.
  • It works in a fresh $RUNNER_TEMP directory that an always() step removes.
  • Only main's own dispatch gets an owned pick.
  • The same minis already run cmux-next.yml's app tests on feat-cmux-next code.

No real-app rehearsal before merge. Downloading the unsigned DEV artifact needs actions: read, which fast-checks commands don't get, and heavy local runs are not allowed.

Why the risk is bounded:

  • publish and tip both need verify to succeed with verified == 'true', and verify runs Apple's codesign --verify --deep --strict on a Mac. A bad Linux signature therefore costs a missed publish, never a bad build.
  • The first dispatched run after merge is watched. The workflow is reverted if sign or verify is red, or if DEV has not published within 30 minutes of merge.

Ruling

The coordinator's ruling: "yes, the verify job may run on owned glaeda minis with the Blacksmith 6-vCPU fallback, within these limits: verify job only: no secrets, no signing key, no publish token in that job; a fresh per-job workspace, removed after the job; rule 6 is narrowed to allow exactly this one job's label set, not widened in general, and the CI test that enforces rule 6 is updated to match; the PR body quotes Leo's rule and names the risk."

Rule 6 as it now reads in tests/test_ci_signing_keychain_is_deleted_before_bundle_code.py:

  1. A job that holds signing, notarization or release-publishing material (or runs in a release environment) runs no bundle code at all, not only outside the keychain window: a process that bundle code leaves running survives into the later steps of the same job. Bundle code runs in jobs without secrets on literal ephemeral runner labels: a job without that material that runs bundle code on what a release job made (it needs one, directly or through others) runs on a literal label, alone or after the fork short-circuit. The one exception is OWNED_VERIFY_JOBS: exactly the job and runs-on named there, which may take an owned Mac free now, holding no secret, outside any environment, not continue-on-error, in a fresh work directory it removes in an always() step. The notarization scripts run in such jobs only with their launch smokes deferred (CMUX_DEFER_LAUNCH_SMOKES) or skipped. STEP_LEVEL_ONLY lists the jobs that still rely on rule 1 alone.

The test enforces this: check_verify_runners() covers the whole tree, and self_test_verify_runners() refuses each of these:

  • another runs-on for the allowed job;
  • the allowed runs-on in another workflow, or on another job;
  • a secret in the allowed job;
  • a missing always() removal;
  • continue-on-error.

Numbers

Before: the last 4 successful publishes (runs 40285252357, 40248339313, 40243744618, 40239171524):

JobRunnerQueueRun
signBlacksmith Mac290-655 s142-181 s
verifyBlacksmith Mac, advisory135-2417 s42-47 s
tipBlacksmith Mac181-3475 s37-300 s
publishLinux, needs sign only1 s12-16 s

After: Linux sign (seconds of queue, about 3-4 min run), then verify on an owned mini when one is free (about 1 min) or Blacksmith 6 vCPU, then publish. That is one fewer Blacksmith macOS job per publish, and Apple's verifier becomes a gate rather than advisory. The real after-numbers come from the first dispatched run, which I will post on this PR.

Tests

On the exact tree: tip feed, keychain/rule 6 (14 jobs), dispatcher, test_dev_app_linux, test_verify_signed_dev_app, test_rcodesign_sign_bundle_dev, production secrets, runner label policy (33), owned-pool rescue, owned-pool sources, the self-hosted guard and YAML parsing all pass.

Agent session: agent session

🤖 Generated with Claude Code