Check that the shipped devbox image is reachable, only when the pair changed

Merged
LA
2mo ago
+286-0

The bake smokes the cmux-tui daemon once, at the moment an image is made. Nothing asks the question that matters later: can a client reach the daemon in the image we are shipping right now, using the cmux-tui build we are shipping right now?

Those two drift apart with no commit at all. files.cmux.com moves to a new client on every cmux-tui release, while the manifest keeps pinning yesterday's snapshot, so the pair in production is one nobody ever tested together.

bun scripts/check-devbox-image-reachable.ts boots the manifest's current default and runs a real round trip, not a port probe: enroll a device, connect over ws://[::1]:1337/v1/link, open a workspace, run a process, read its output back. When the live client differs from the baked one, it installs the live binary in the guest and repeats the entire trip with it, so a protocol change that would break a freshly built app fails here instead of in front of a user.

Measured on the current default: 17s with both legs, 9s when the live client is already the baked one, one sm VM, deleted on every path including failure.

Only when necessary

  • Path-filtered to the manifest, the devbox source, the client resolver, and the check itself.
  • A daily schedule, because the client moves without a commit.
  • The run keys a cache on every default image id plus the live client's sha256 and skips a pair that has already passed, so the daily run costs nothing on a day when neither moved. Only a pass is recorded, so a failure re-runs.
  • Fork PRs skip cleanly: booting a machine for one would hand an untrusted branch the production account.

What it needs

FREESTYLE_API_KEY as an Actions secret. Without it the job says so and skips rather than failing, so this can land before you decide on that. It is the same decision as the bake-on-main job we discussed; this one is a much smaller blast radius, since it only ever boots one small VM and deletes it.


View with [code]smith Autofix with [code]smith
Need help on this PR? Tag @codesmith-bot with what you need. Autofix is disabled.


[!NOTE]
Medium Risk
CI boots real Freestyle VMs with a production API key (mitigated by environment scoping, fork skips, and single small VM lifecycle), but failures could block merges on external client/image drift.

Overview
Adds cloud VM image reachability CI that boots the manifest’s default devbox snapshot and proves the shipped cmux-tui client can complete a full daemon round trip (not just a port check). When the live client from files.cmux.com differs from what’s baked in the image, the check downloads the live binary in the guest and repeats the trip so image/client drift is caught before users hit it.

A new check-devbox-image-reachable.ts script drives Freestyle VM create/exec/delete, exposes reachabilityKey (default image ids + live client sha256) for cache keys, and supports --print-key so CI can skip unchanged pairs. The workflow runs on relevant path changes, daily (client can move without a repo commit), and workflow_dispatch; it gates on FREESTYLE_API_KEY (skip on fork PRs or missing secret), uses Actions cache so only new pairs boot a VM, and records passes only so failures re-run. Unit tests cover reachabilityKey stability and invalidation when images or client pins change.

Reviewed by Cursor Bugbot for commit 95a83a503c67552f4c43e1f3d2ec33943a76b867. Bugbot is set up for automated code reviews on this repo. Configure here.


Summary by cubic

Adds a CI job that boots the shipped devbox image and proves the shipped cmux-tui client can reach the daemon inside it, so the image/client pair in production is what actually gets tested.

  • Runs a real round trip (enroll, connect over ws://[::1]:1337/v1/link, open a workspace, run a process, read output), not a port probe.
  • When the live client differs from the baked one, installs the live binary in the guest and repeats the trip.
  • Triggered on path changes, a daily schedule, and manually; a cache key over image ids plus client sha256 skips pairs that already passed.
  • Costs one small sm VM, deleted on every path including failure; ~17s with both legs, ~9s when the client is already baked.
  • Requires FREESTYLE_API_KEY; without it the job skips rather than failing, and fork PRs skip without a credential.
  • A temporary override to a stock image proved the job fails cleanly and was reverted, so the check targets the manifest default.