Keep a Cloud machine's edge credential valid for the machine's life

Merged
LA
11d ago
+139-21

About 95% of guest polls to /api/vm/reflection/name answer 401: 41k per 30 minutes, 2.0M on 2026-10-06. The 401 count was ~100k a day until 2026-10-02 and jumped on 2026-10-03 (637k, then 951k, 1.75M, 2.02M), exactly 30 days after the Freestyle production cutover.

Cause: a machine's coderouter credential is injected by the provider edge from a rule fixed at create (provisionVmModelPlane; Freestyle cannot change TLS rules on a running guest). Both token schemes expire after 30 days: legacy machine-bound crt_ rows by expires_at, signed x-cmux-authorization JWTs by exp/maxTokenAge and expires_at. Nothing renews them, so every machine older than 30 days lost coderouter (its agents' model access) and the reflection API, and its guest kept polling with a dead credential. Signed tokens (2026-09-22) would start failing the same way on 2026-10-22.

Fix: a machine-bound credential lives as long as its machine. verifyVmAuthorization checks the time claims as of min(now, exp), so signature, key, audience, type, a future iat and a window longer than the lifetime still fail. The repository drops the clock condition for machine-bound tokens only; every request still requires the row not revoked, bound to the claimed machine and owner, and the machine provisioning/running/paused. Unbound CLI sessions still expire.

Measurability: refusals now set cmux.coderouter.auth_failure and cmux.coderouter.auth_failure_detail (placeholder_only, no_credential, signed_unverified, not_live, binding, chatmux_unverified) on the route span, so the remaining ~100k/day baseline can be split by cause in Axiom.

Red/green on cmux-lawrence-2 at d8a7fe5 vs head: tests/coderouter-vm-authorization.test.ts, tests/coderouter-route-token-repository.test.ts, tests/coderouter-request-telemetry.test.ts. Every coderouter*, vm-reflection*, vm-principal* test file passes individually; vm-guest-prompt.test.ts has one OSC 7 failure that also fails on main on that host. bun run typecheck, bun run lint:complexity, eslint on changed files pass.

Security note: the clock no longer ends a machine credential; destroy and team-member removal revoke it, and the live-machine join runs on every request. The token never enters the guest (edge header values read back as ***).


View with [code]smith Autofix with [code]smith
Need help on this PR? Tag @codesmith-bot with what you need. Autofix is disabled.


[!NOTE]
High Risk
Changes authentication and JWT verification for VM-bound credentials; security still depends on revocation, binding, and live-machine checks rather than clock expiry.

Overview
Fixes 30-day 401 storms on long-running Cloud VMs by treating machine-bound coderouter credentials as valid for the machine’s lifetime, since edge-injected tokens cannot be renewed on a running guest.

JWT path (verifyVmAuthorization): time checks use verificationInstant (min(now, exp−1s)), so past-exp tokens still verify cryptographically while future iat, oversized windows, and other bad claims still fail. DB path (authenticateRouteToken / authenticateVmAuthorization): drops expires_at for VM-bound rows; legacy crt_ tokens use vm_id IS NOT NULL OR expires_at > now so unbound CLI sessions still expire. Revocation, binding, and live VM status checks remain on every request.

Observability: failed auth now sets span attributes cmux.coderouter.auth_failure and cmux.coderouter.auth_failure_detail (placeholder_only, no_credential, signed_unverified, not_live, binding, chatmux_unverified) via new RouteTokenAuthFailureDetail on RouteTokenAuthResult.

Reviewed by Cursor Bugbot for commit ec56e613f98370272b6a53ce377066cbcebd342b. Bugbot is set up for automated code reviews on this repo. Configure here.


Summary by cubic

Fixes Cloud machines losing coderouter and the reflection API 30 days after creation. The provider edge injects the machine credential from a rule fixed at create and both token schemes expired, so guests kept polling with a dead credential—401s grew from ~100k to 2M per day.

Behavior

  • Machine-bound credentials now live as long as their machine, checked on every request against the database row: not revoked, bound to the claimed machine and owner, machine live.
  • Unbound CLI sessions still expire, and signed tokens still fail on invalid signature, a future iat, or an overlong window.

Observability

  • Auth refusals record a failure detail on the route span (cmux.coderouter.auth_failure_detail) so the remaining baseline can be split by cause in Axiom.