Cloud VM image: production channel (promote channel production, prod-copy bake), inert until its environment exists

Merged
TE
8d ago
+1106-51

Production stays inert until Leo creates the GitHub environment cmux-next-vm-image-prod (branch feat-cmux-next only; secrets FREESTYLE_API_KEY for the production account, CMUX_UITEST_STACK_EMAIL, CMUX_UITEST_STACK_PASSWORD; and variable FREESTYLE_PROD_ANCHOR, <slug> <sh-id> of a snapshot on the production account, until a production snapshot is recorded). Until then, every production run stops in its first job with a clear error. That job names no environment, so nothing is created or changed. The dev channel behaves as before.

Leo, 2026-10-28: "whatever's convenient for cloud, like that can be productionized, operationalized, we ought to just put out there"

What changed

The Cloud VM image workflows can now promote a production image. Today they can only promote dev.

  • cloud-vm-image-promote.yml takes a channel input: dev (the default) or production.
    • The promote job's environment follows the channel: production runs in cmux-next-vm-image-prod, dev stays in cmux-next-vm-image-dev.
    • Production takes a prod-copy bake run as bake_run. It records that run's channel-production.json as names.production on the dev image's history entry (promote-record.ts --production-copy).
    • It then runs promote.ts --channel production. promote-lib.ts already handles production: it runs the cmux-old compat gate as the test profile (CMUX_UITEST_STACK_*, production only) and smokes the copy on the production account.
    • The one change to promote-lib.ts: the image smoke's child process (runImageSmoke) no longer inherits the compat gate's test profile or the GitHub token (smokeEnv). In the promote step, those values leave the environment before any command runs, and only promote.ts gets them.
    • The diff artifact also carries channels/production.json. On the first promotion it is a new file, added with git add -N so the diff includes it. The production deploy after the channel PR lands ships the image.
    • delete_snapshot is refused for production. Its guard deletes dev-account images only.
  • cloud-vm-image-bake.yml has a new mode, prod-copy, with input dev_snapshot.
    • It bakes the production copy of a dev image already recorded in channels/dev.json history, in cmux-next-vm-image-prod.
    • prod-copy.ts plan reuses the dev image's flags (--agent-tools, --team-vm). It refuses when the lock differs from the one the dev image recorded, for a dev tree bake, for a non-CI or unsmoked entry, and for an entry that already has a production copy.
    • The copy is named cmuxnp-prod-vmimg-<tag>, the prefix the production Worker boots (bake.ts --prod-copy). It is smoked, and kept only once its candidate has been written.
  • The account check (prod-copy.ts account-check) runs before any production bake or promote. It uses read-only GETs.
    • It refuses the dev account's key: a key that resolves the dev channel's snapshot name to its recorded id belongs to the dev account.
    • Once a production snapshot is recorded (channels/production.json or a names.production), the key must also resolve one of those snapshots to its recorded id, so a third account's key is refused as well.
    • The very first production copy has no recorded snapshot to identify the account by. For it, the environment names one: the variable FREESTYLE_PROD_ANCHOR (not a secret), <slug> <sh-id> of any snapshot on the production account, set next to the key. With neither an anchor nor a recorded production snapshot, the check refuses.
  • In the prod-copy steps, the key reaches bun only as a 0600 file (FREESTYLE_API_KEY_FILE, removed when the step ends), never in a child process's environment, the same as the promote steps. The prod-copy run has its own cleanup step, so the dev cleanup step is unchanged.

Fail closed

  • A prod-gate job runs first for production promotes and prod-copy bakes.
    • It names no environment. A job that names a missing environment would create it with no protection, and repo or org secrets would then flow into it.
    • It reads the environment through the API, which works with the existing read-only token. It requires the environment to exist with a custom branch policy that is exactly branch:feat-cmux-next. Otherwise it stops with "production stays off until cmux-next-vm-image-prod exists …".
    • The production job only runs when the gate succeeded. Dev skips the gate and runs exactly as before (needs plus !cancelled() && (success || dev && skipped)).
  • Inside the job, missing secrets refuse the run (prod-copy bake and production promote alike). FREESTYLE_API_KEY, CMUX_UITEST_STACK_EMAIL and CMUX_UITEST_STACK_PASSWORD are checked for presence only (secrets.X != ''), never by value, and a missing one refuses before checkout or any provider call.
  • No permissions: change (promote: contents: read and actions: read; bake: contents: read), no pull_request_target, and no secrets were set. This PR only adds references; Leo places the values.

Risk

  • Dev: the dev steps' scripts gained only CHANNEL branches. For dev, the new step env values evaluate to empty, and the promote step unsets them before promote.ts runs.
  • Production can't run until the environment exists. A production promote still produces only a diff artifact; the channel PR and the production deploy remain separate, reviewed steps.
  • The production copy carries --agent-tools when the dev image had it. Until now, bake.ts allowed --agent-tools only for dev snapshots. A copy has to be the image that was tested on dev, so --prod-copy allows it. --dev-tree stays refused.
  • The copy is baked from the workflow's commit with the dev image's lock and flags, so changes to the bake code since the dev bake apply. The copy's own smoke and the promote smoke cover that.

Tests

  • New web/tests/vm-image-cmux-vm-prod-channel.test.ts:
    • production uses cmux-next-vm-image-prod and dev keeps cmux-next-vm-image-dev, in both workflows;
    • the gate is identical in both workflows, and is run with a stubbed gh for a missing environment, no policy, extra branches, a tag policy and a wrong ref;
    • prod-copy.ts plan refusals and the account check;
    • promote-record.ts --production-copy;
    • bake.ts --prod-copy naming and refusals;
    • the production promote step's key handling: a 0600 file only, the account check before promote.ts, and dev getting no production values;
    • the diff including a new production.json.
  • Updated tests:
    • vm-image-cmux-vm-promote-workflow.test.ts: trust boundary, secret references, input checks for production;
    • vm-image-cmux-vm-bake-workflow.test.ts: the prod-copy steps run with a stubbed bun;
    • tests/test_supplychain_security.py: both environment expressions, and no environment on the gate jobs.
    • scripts/cmux-next/release/test/promote.test.ts: smokeEnv.
  • Run on the Linux pool (cmux-ci test): the four web/tests/vm-image-cmux-vm-* files (158 pass), bun run typecheck in web, the release-rails promote.test.ts (15 pass) and its tsc, eslint on the changed files (0 errors), actionlint on both workflows, and the supply-chain environment test (run locally).
  • Nothing was dispatched: no bake, no promote, no production run.

Size: about 1,160 changed lines. More than half are tests, and the two workflows depend on each other's artifact (channel-production.json), so they aren't split.

Agent session: agent session

šŸ¤– Generated with Claude Code