feat(api): trace a cloud workspace from start to host-service answering

Merged
SA
6d ago
+263-43

Summary

  • A cloud workspace's start and its provision job are now one Sentry trace, from cloudWorkspace.create until host-service on the box answers.
  • sandbox.fork gets its own span, tagged with source golden, region, vCPUs and sandbox name, so we can report its latency to Vercel apart from our own work.
  • The box's own boot phases (which superset-boot and host-service already stamp and serve on health.check) are recorded as spans at the box's timestamps.

Why / Context

We want to tell our sandbox provider how long sandbox.fork takes, and to see where the rest of a box's startup goes. Over the last 30 days (~930 forks) the fork call has been most of the job: p50 5.6 s, p90 7.2 s, p99 10.2 s, out of a job p50 of 8.8 s. Its latency is also bimodal: about a third finished under 2 s (p50 0.75 s), and the rest took a p50 of 6.1 s. Today that call is hidden inside the create span, the QStash delay before the job is not measured, and the box's boot stamps are never read on the API side.

How It Works

cloud-workspace.start              ← new; every caller of startCloudWorkspace
  (gap = QStash delivery)
cloud-workspace.provision          ← existing job, now in the same trace
  ├─ claim
  ├─ create
  │   ├─ fork | create from image  ← new
  │   └─ start boot                ← new: identity, sign-in files, boot command
  ├─ settle
  └─ box boot                      ← new: one span per boot stamp, at the box's own times
  • startCloudWorkspace wraps its body in a forced transaction, the same way provisionCloudWorkspace already does. It flushes through waitUntil, so the create response is not delayed.
  • The API sampler accepts exactly cloud-workspace.start and cloud-workspace.provision. Request tracing stays off.
  • publishCloudWorkspaceJob sends sentry-trace/baggage as publish headers. QStash forwards them (the SDK prefixes them Upstash-Forward-), and the provision route calls Sentry.continueTrace, which makes the start span the job's parent.
  • waitForHostService now returns the stamps from the first health.check that answers. A body it can't parse still counts as answered, so the polling behaves as before.
  • boot-trace.ts turns stamps into segments, each running from one stamp to the next. Segments take their names from the stamps rather than from a list of phases, so a box on an older or newer boot runner still traces. A host-service too old to report its boot just leaves box boot out.

Manual QA Checklist

Not done yet: this needs a deploy to exercise, and I have not deployed it.

  • Create a cloud workspace on a preview or production deploy; in Sentry, cloud-workspace.start and cloud-workspace.provision share one trace id, with provision under start
  • fork span shows under create with sandbox.* attributes
  • box boot shows the stamped phases ending at host.listening → host answered
  • A create through the CLI traces the same way
  • Waking an existing workspace is unchanged

Testing

  • bun test src/lib/sandbox/ src/router/cloud-workspace/ in packages/trpc: 72 pass, including the new boot-trace.test.ts (health body parsing, segment mapping, clamping when our clock trails the box's)
  • tsc --noEmit in packages/trpc, apps/api, packages/sandbox (whose release scripts import vercel.ts)
  • biome check on the changed files

Known Limitations

  • Stamps are read when host-service first answers, so the later boot phases (desktop, checkout) are not in the trace. They do not block host-service being responsive.
  • The trace ends at host-service answering, not at a terminal responding in the desktop. That part of the chain (row seen as ready, access ticket, terminal dial, pty-daemon, shell) is not traced.
  • Box spans use the box's clock and the API spans use Vercel's. Both run NTP, so any skew should be small; the last segment is clamped so it never goes negative.

Risks / Rollout

  • Risk: low. This only adds spans, plus headers on QStash publishes. Other job routes don't continue the headers, and the sampler still rejects their names.
  • Volume: about two transactions per create (~1k creates a month).
  • Rollback: revert the commit. There is no data or contract change, and no box, bundle or desktop change.

Summary by cubic

A cloud workspace's start and its provision job are now one Sentry trace, from the create call until host-service answers on the box. Until now the fork call was hidden inside the create span, the QStash delay was unmeasured, and the box's boot stamps were never read; the fork call has been most of the job (~930 forks, p50 5.6 s of a p50 8.8 s job), so it is now reported to the sandbox provider as its own span.

Trace shape

  • cloud-workspace.start wraps every startCloudWorkspace call; its span flushes through waitUntil, so the create response is not delayed.
  • The provision job continues the start's trace: publishCloudWorkspaceJob sends sentry-trace/baggage through QStash's forwarded headers, and the provision route calls Sentry.continueTrace.
  • fork (or create from image) and start boot are their own spans under create; the fork span is tagged with source, region, vCPUs, and sandbox name.
  • waitForHostService returns the boot stamps from the first health.check that answers, and recordBootSpans turns them into box boot spans at the box's own timestamps, named from the stamps so older and newer boot runners still trace.

Behavior notes

  • Recording the boot spans is best-effort: a failure is logged, and provisioning continues without deleting or stopping the box.
  • The sampler accepts exactly the two transaction names; request tracing stays off.
  • A health.check body that can't be parsed still counts as answered, so polling behaves as before.
  • The trace ends where host-service answers; the box's later phases and the terminal chain are not traced.
  • No tRPC input or output, bundle, image, or desktop changes; rollback is a revert.

Summary by CodeRabbit

  • Improvements
    • Cloud workspace startup and provisioning now include more detailed timing information for boot stages, making startup activity easier to inspect.
    • Trace information is carried across workspace startup and provisioning, helping show these steps together.
    • Startup checks can retain available boot-stage details when optional response data is incomplete; existing retry and timeout behavior remains unchanged.