cloud: a headless NEXT Cloud journey on previews, staging and production deploys

TE
2d ago
+746-0

Leo asked (2026-11-04):

let's also get our iteration loop good for the whole "working on cloud" type thing

Getting a NEXT Cloud chat working end to end took days, because every check waited on an app build or a nightly. This PR adds one command that checks a backend end to end with no app build, and runs it on every PR preview and after every staging and production deploy.

What changed

  • scripts/cmux-next/cloud-journey.ts: bun scripts/cmux-next/cloud-journey.ts --backend local|dev|preview-N|staging|prod (or --api URL --stack dev|prod). It signs in as the agent test account and walks the routes the app and CLI use on the cmux API Worker:

    • health, sign in, user.ensure, install token
    • create machine, running, listed, first exec
    • start agent (cmux-tui acp ensure on the machine), send chat, reply
    • connect (connect_info + link_token), handoff record, a folder round trip
    • pause, resume, exec after resume, delete

    Each step prints PASS/FAIL/SKIP and its latency. --out DIR writes journey.json and summary.md. The machine it made is deleted at the end, pass or fail. --team runs as another team, and --cmux PATH sends every exec through cmux machine access exec, so the journey also checks the CLI. A failed machine fails its step at once, with the row's error.

  • .github/workflows/backend.yml:

    • journey-preview runs after the preview job, against cmux-api-preview-<PR>. Its step table is the job summary, and a failed step makes that PR's check red.
    • journey-staging and journey-production run after each deploy (including promote). They run as the test team and, when CMUX_JOURNEY_PRODUCT_TEAM is set, as a product team on the never-pause path.
    • Secrets: staging reads the pair cloud-chat-proof.yml already uses (cmux-next-vm-image-dev). Preview and production read cmux-next-cloud-journey (CMUX_JOURNEY_DEV_STACK_*, CMUX_JOURNEY_PROD_STACK_*). Without them, the job posts a notice and passes.
    • The journey script's own paths trigger the workflow.

  • scripts/cmux-next/cloud-dev-loop.sh --tag NAME [--platform-checkout DIR]: the inner loop. It deploys this worktree's API Worker as your own cmux-api-preview-NAME and, if given, a cmux-platform worktree as cmux-platform-preview-NAME. Then it runs the journey there and prints the seconds for each part.

  • backend/scripts/deploy-worker.sh: CMUX_PREVIEW_PLATFORM_WORKER=cmux-platform-preview-<n> points a preview's PLATFORM service and CLOUD_DO at a platform preview instead of development.

  • --app-cli PATH --socket SOCK runs the commands a person runs, through a running NEXT build's own CLI: cloud list-machines, vm token create, cloud new-machine, machine-status, exec, start-session, start-chat --wait, and kill-machine (always, at the end). It is a mode of the same script rather than the scripts/cloud/cli-journey.sh the issue sketched, so each step gets the same timing and table. The backend named per step comes from the answer when the answer gives one.

Closes #35086.

First result

Production, from a team Mac, as the production test account: health, sign in, user.ensure, install token and create passed (55 ms to 1.8 s each). Then the machine went to failed after 34 s, both times, with cloud.provider.unavailable: VM API: snapshot capture is still being confirmed. That is a live production bug that no existing smoke catches, and it is handed to its own lane in cmux-platform.

Notes

  • The repo note says not to add CI jobs. Leo asked for exactly these: the journey on previews and as the deploy smoke.
  • The preview journey sends the dev Stack test account's short-lived session to the PR's own preview Worker. That happens only on same-repo PRs, which the preview job already limits to, and with a test account that holds no data.

Observed / Not run

  • Ran: two production journeys (steps above); actionlint and a YAML parse of backend.yml; prepush-gate.
  • Not run here: cloud-journey.test.ts (sent to the fleet), and the staging, preview and local backends. This host has no dev-project pair, so those first run in CI.

Changelog

none

Agent session: agent session