feat(runs): durable cloud execution and Freestyle dogfood

Merged
TE
19d ago
+1621-427

What changed

Adds the first production slice of the Cloud Run control plane: durable team-scoped Runs and attempts, fenced worker leases, cancellation and replay-safe checkpoints, exact source/setup receipts, attach tickets, event/artifact replay, retained VM promotion, and a protected cron recovery path.

The managed dogfood now creates an asynchronous Run, polls its durable receipt, verifies the exact GitHub commit and .cmux/setup.json digest, reuses the retained Freestyle VM for the terminal journey, checks attach-ticket expiry and single-use redemption, exercises promotion, and records cleanup. A daily staging-only dogfood schedule is included; Huge Blue remains fail-closed.

Validation

  • bun run typecheck
  • targeted ESLint for Run routes/services/tests
  • bun run lint:complexity
  • source/setup, worker-contract, and dogfood probe tests
  • live PostgreSQL behavior suites: 8 passed, 0 failed, 34 expectations (leases, expiry/replay fencing, cancellation, bounded concurrency, team idempotency/isolation, immutable artifacts, attach TTL/single-use)
  • actionlint .github/workflows/cloud-vm-dogfood.yml

The actual Freestyle dogfood requires deployment of this exact head; the workflow is ready to run against staging after the PR is deployed.


View with [code]smith Autofix with [code]smith
Need help on this PR? Tag @codesmith-bot with what you need.


Summary by cubic

Adds durable team-scoped Runs and attempts backed by a protected cron worker, replacing the synchronous user-scoped flow. The Freestyle dogfood journey now creates an asynchronous Run, verifies the exact GitHub commit and .cmux/setup.json digest, and reuses the retained VM for the terminal session.

Key changes

  • Runs are team-scoped with per-team idempotency; new partial unique indexes leave legacy user-scoped rows untouched.
  • Attempts claim fenced leases, and cancellation, checkpoints, and terminal transitions go through a shared state-machine guard with replay-safe events.
  • Attach tickets are hashed, single-use, and expire; retained VMs can be promoted into the existing Cloud machine lifecycle.
  • A new migration adds owner_team_id, run spec JSONB, cancellation event kinds, and attempt lease/retention columns.

Validation

  • Typecheck, targeted ESLint, complexity lint, and contract/dogfood probe tests pass.
  • Live PostgreSQL suites pass 8 tests with 34 expectations covering leases, expiration and replay fencing, cancellation, bounded concurrency, team isolation, immutable artifacts, and attach TTL/single-use redemption.
  • The dogfood workflow runs against staging after this head is deployed; Huge Blue stays fail-closed.

Summary by CodeRabbit

  • New Features

    • Added durable, team-scoped Runs with asynchronous execution, status and event tracking, and reusable request handling.
    • Runs can verify a GitHub commit and setup manifest, execute setup steps, and retain results and artifacts.
    • Added controls to cancel Runs, attach to eligible attempts, and promote successful attempts for continued use.
    • Added scheduled processing for queued Runs.
  • Bug Fixes

    • Improved handling of duplicate requests, invalid inputs, expired attachment tickets, and conflicting result updates.