feat(sandbox): box log shipper, off by default (M6)

Merged
SA
21h ago
+1037-72

What and why

M6 of the cloud workspace cost and health plan: a small shipper that sends the platform's own box logs to Sentry Logs, so a box that failed to start or was killed can be read without waking it.

  • superset-logship (new, bundle rootfs, bash + jq + curl). Reads boot.log, memguard.log, desktop.log and host-service.log from a byte offset per file, saved under /var/lib/superset/logship/. It never reads start-hook.log or checkout.log. Record contract (the admin API reads these): attributes log.source, cloud_workspace.id, organization.id, log.file.offset, box.boot_started_at (epoch ms of the last boot.start at or before the line), log.repeat_count (only when above 1); plus body, the time parsed from the line, level info. A line with no time takes the time and boot of the line before it. About 506 bytes per record.
  • No history: on first enable each file starts at its first line of the current boot. The shipper trusts saved offsets only if it ran in the current or the previous boot; otherwise it starts again at the current boot. So off, then on, never sends what was written while off.
  • Controls:
    • Drops [host-service:vitals] and memguard rank lines; memguard kills are kept.
    • Reads only regular files; a symlink in the log directory is skipped.
    • Scrubs token formats.
    • Collapses repeated lines into one record with log.repeat_count.
    • Caps the serialized size of records per box per hour (256 KB, a literal in the script) and sends one log shipper: cap of … reached record, counted in the cap; nothing more is sent that hour.
    • Collapses repeated lines only within one pass (one read of up to 64 KB of one file).
    • A refusal from Sentry (403, 429, 5xx, network) waits and buffers nothing.
  • superset-boot starts it as the sandbox user (setpriv, from a state directory it creates for that user) after memguard, only when sandbox.conf has SUPERSET_SANDBOX_BOX_LOGS=1 and a DSN, with nice 10, without the host secret.
  • Hook commands never logged (every box): the runner used to copy host-service's whole runStartHook answer into boot.log, including the hook command and, on an early failure, its output tail. host-service also printed every hook command into host-service.log, so a multi-line command left lines 2..n there. Now host-service prints [sandbox] start hook running (pid N) (command of L chars) and the like (start and archive hooks, four places, through hookOutcomeLine). It stamps hook.exited exit=0 ms=… or hook.failed exit=N ms=… into boot.log from the setup child's exit handler. The runner stamps only hook.started pid=…, hook.none, hook.start.failed or hook.skipped, and does not poll.
    • Two limits a reader of boot.log must know. The exit stamp is written only by the host-service that spawned the setup, so a setup that outlives a host-service restart gets no exit stamp. A setup that fails within 3 s writes two lines, hook.start.failed (runner) and hook.failed exit=N ms=N (host-service); count them as one failure. The release rebuilds host-service from the same checkout, so both sides change together.
  • One flag helper: organizationFlagEnabled(flags, flag, organizationId) and boxLogsEnabled(flags, organizationId) live in lib/sandbox/feature-flags.ts and take the PostHog client. After #9929 reduced agentToolsDriveEnabled to one PostHog call, the claim asks sandbox-agent-tools-drive through organizationFlagEnabled too, and lib/sandbox/agent-tools.ts and its test are removed.
  • Size: 19 files, +1037 / −72 against main. Most of it is the new script (271) and its tests (482); the other 17 files are +284 / −72. Details on the evidence page.
  • Switches: two PostHog flags, both failing closed: sandbox-box-logs-fleet (fleet switch, a plain isFeatureEnabled(flag, "sandbox-fleet") with no group, asked first) and sandbox-box-logs (per organization, asked only while the fleet switch is on). The claim calls env.SENTRY_DSN_SANDBOX ? boxLogsEnabled(posthog, orgId) : false beside its other first lookups; feature-flags.test.ts tests both-on and fleet-off-asks-once. buildSandboxClaim adds SUPERSET_SANDBOX_BOX_LOGS="1" only when both are on and the API has SENTRY_DSN_SANDBOX. These are additive optional keys in sandboxIdentitySchema: no contract version bump and no tRPC change.

What breaks, and for whom

Nothing is expected to break. Merging releases the bundle to the fleet, and every box takes it at its next wake.

  • With either flag off (the default), a box behaves as today, except that boot.log hook stamps now carry an exit code and a duration in place of host-service's raw answer. Anything that greps hook.started {"started":true… would need to change; I found no reader in the repo.
  • Turning a flag on or off takes effect at a box's next wake, because sandbox.conf is written only at claim and wake.

How it is switched on

  1. Both flags are off on merge. Do not turn either on for a customer organization until the policy wording (L1) is live.
  2. Create sandbox-box-logs-fleet in PostHog with 100% rollout. It is a kill switch: set it to 0% and every box stops shipping at its next wake.
  3. Create sandbox-box-logs with an organization group condition, and add organizations one at a time.

First real box (after merge)

  1. Wait for production-release to finish (Sandbox workflow on main), so environments point at this bundle.
  2. In PostHog, turn on sandbox-box-logs-fleet (100%) and sandbox-box-logs for our own organization only (group condition on its id).
  3. Stop and wake one of our cloud workspaces, or create one. The claim at wake writes SUPERSET_SANDBOX_BOX_LOGS='1' into /etc/superset/sandbox.conf.
  4. On the box:
    • grep logship /var/log/superset/boot.log shows logship.start pid=….
    • cat /var/log/superset/logship.log is empty, or shows one refused: <code> line for each refusal.
    • ls /var/lib/superset/logship/ shows one .offset file per source.
  5. In Sentry, open the sandbox project → Logs, and search cloud_workspace.id:<id>. Expect:
    • boot lines from the current boot.start on, with log.source=boot and box.boot_started_at set (epoch ms).
    • host-service lines with their own ISO times.
    • No [host-service:vitals] lines, and no memguard rank lines.
    • [sandbox] start hook running (pid N) (command of L chars), and no line of the hook command anywhere.
    • hook.exited or hook.failed with exit= and ms=.
  6. Check what does not arrive: no text from start-hook.log or checkout.log.
  7. Check the volume after one awake hour: about 40–50 records for a quiet box. A log shipper: cap of … reached record means the cap needs a look.
  8. Turn our organization's flag off, wake the box, write a line, then turn it on and wake again: that line must not arrive.
  9. Boot time: in boot.log, compare boot.start to host.ready on a wake with the flag on and a wake with it off. This was not measured before merge.
  10. If logship.log shows refused: 403, the sandbox project does not take logs. Turn the fleet switch off and settle it with M7.

Evidence

Checks

  • packages/sandbox: bun test (84 pass, including 15 in logship.test.ts), bun run build, bun run typecheck
  • packages/trpc: bun test src/lib/sandbox (47 pass), bun run typecheck
  • packages/shared: bun test (1692 pass), bun run typecheck
  • packages/host-service: bun test src/runtime/sandbox-self-seed src/runtime/boot-stamps (34 pass), bun run typecheck
  • bun run lint (the repo lint script, as CI runs it): clean
  • turbo typecheck --concurrency=1 across the repo (CI's concurrency 3 runs out of memory on the workspace VM)
  • The boot-twice job on this PR has a new check: with the flag off, no shipper stamp, process or state.

Not verified: a real cloud box, and whether the sandbox Sentry project accepts log envelopes (see the evidence page).

Summary by CodeRabbit

  • New Features
    • Sandbox box logs can now be sent to Sentry when enabled for both the organization and the fleet.
    • Log delivery excludes setup scripts and clone output, filters selected entries, redacts credential patterns, and limits the amount of data sent.
  • Improvements
    • Sandbox hook messages report outcomes without displaying the hook command text.
    • Boot logs now distinguish hook outcomes, including successful starts, failures, and cases where no hook runs.
  • Documentation
    • Added guidance on box-log delivery and the information it may contain.