feat(sandbox): box log shipper, off by default (M6)
Merged
SA
21h ago+1037+1037+0123456789012345678901234567890123456789-72-72-01234567890123456789
What and why
M6 of the cloud workspace cost and health plan: a small shipper that sends the platform's own box logs to Sentry Logs, so a box that failed to start or was killed can be read without waking it.
superset-logship(new, bundle rootfs, bash + jq + curl). Readsboot.log,memguard.log,desktop.logandhost-service.logfrom a byte offset per file, saved under/var/lib/superset/logship/. It never readsstart-hook.logorcheckout.log. Record contract (the admin API reads these): attributeslog.source,cloud_workspace.id,organization.id,log.file.offset,box.boot_started_at(epoch ms of the lastboot.startat or before the line),log.repeat_count(only when above 1); plusbody, the time parsed from the line, levelinfo. A line with no time takes the time and boot of the line before it. About 506 bytes per record.- No history: on first enable each file starts at its first line of the current boot. The shipper trusts saved offsets only if it ran in the current or the previous boot; otherwise it starts again at the current boot. So off, then on, never sends what was written while off.
- Controls:
- Drops
[host-service:vitals]andmemguard ranklines; memguard kills are kept. - Reads only regular files; a symlink in the log directory is skipped.
- Scrubs token formats.
- Collapses repeated lines into one record with
log.repeat_count. - Caps the serialized size of records per box per hour (256 KB, a literal in the script) and sends one
log shipper: cap of … reachedrecord, counted in the cap; nothing more is sent that hour. - Collapses repeated lines only within one pass (one read of up to 64 KB of one file).
- A refusal from Sentry (403, 429, 5xx, network) waits and buffers nothing.
- Drops
superset-bootstarts it as the sandbox user (setpriv, from a state directory it creates for that user) after memguard, only whensandbox.confhasSUPERSET_SANDBOX_BOX_LOGS=1and a DSN, withnice 10, without the host secret.- Hook commands never logged (every box): the runner used to copy host-service's whole
runStartHookanswer intoboot.log, including the hook command and, on an early failure, its output tail. host-service also printed every hook command intohost-service.log, so a multi-line command left lines 2..n there. Now host-service prints[sandbox] start hook running (pid N) (command of L chars)and the like (start and archive hooks, four places, throughhookOutcomeLine). It stampshook.exited exit=0 ms=…orhook.failed exit=N ms=…intoboot.logfrom the setup child's exit handler. The runner stamps onlyhook.started pid=…,hook.none,hook.start.failedorhook.skipped, and does not poll.- Two limits a reader of boot.log must know. The exit stamp is written only by the host-service that spawned the setup, so a setup that outlives a host-service restart gets no exit stamp. A setup that fails within 3 s writes two lines,
hook.start.failed(runner) andhook.failed exit=N ms=N(host-service); count them as one failure. The release rebuilds host-service from the same checkout, so both sides change together.
- Two limits a reader of boot.log must know. The exit stamp is written only by the host-service that spawned the setup, so a setup that outlives a host-service restart gets no exit stamp. A setup that fails within 3 s writes two lines,
- One flag helper:
organizationFlagEnabled(flags, flag, organizationId)andboxLogsEnabled(flags, organizationId)live inlib/sandbox/feature-flags.tsand take the PostHog client. After #9929 reducedagentToolsDriveEnabledto one PostHog call, the claim askssandbox-agent-tools-drivethroughorganizationFlagEnabledtoo, andlib/sandbox/agent-tools.tsand its test are removed. - Size: 19 files, +1037 / −72 against main. Most of it is the new script (271) and its tests (482); the other 17 files are +284 / −72. Details on the evidence page.
- Switches: two PostHog flags, both failing closed:
sandbox-box-logs-fleet(fleet switch, a plainisFeatureEnabled(flag, "sandbox-fleet")with no group, asked first) andsandbox-box-logs(per organization, asked only while the fleet switch is on). The claim callsenv.SENTRY_DSN_SANDBOX ? boxLogsEnabled(posthog, orgId) : falsebeside its other first lookups;feature-flags.test.tstests both-on and fleet-off-asks-once.buildSandboxClaimaddsSUPERSET_SANDBOX_BOX_LOGS="1"only when both are on and the API hasSENTRY_DSN_SANDBOX. These are additive optional keys insandboxIdentitySchema: no contract version bump and no tRPC change.
What breaks, and for whom
Nothing is expected to break. Merging releases the bundle to the fleet, and every box takes it at its next wake.
- With either flag off (the default), a box behaves as today, except that boot.log hook stamps now carry an exit code and a duration in place of host-service's raw answer. Anything that greps
hook.started {"started":true…would need to change; I found no reader in the repo. - Turning a flag on or off takes effect at a box's next wake, because
sandbox.confis written only at claim and wake.
How it is switched on
- Both flags are off on merge. Do not turn either on for a customer organization until the policy wording (L1) is live.
- Create
sandbox-box-logs-fleetin PostHog with 100% rollout. It is a kill switch: set it to 0% and every box stops shipping at its next wake. - Create
sandbox-box-logswith an organization group condition, and add organizations one at a time.
First real box (after merge)
- Wait for
production-releaseto finish (Sandbox workflow on main), so environments point at this bundle. - In PostHog, turn on
sandbox-box-logs-fleet(100%) andsandbox-box-logsfor our own organization only (group condition on its id). - Stop and wake one of our cloud workspaces, or create one. The claim at wake writes
SUPERSET_SANDBOX_BOX_LOGS='1'into/etc/superset/sandbox.conf. - On the box:
grep logship /var/log/superset/boot.logshowslogship.start pid=….cat /var/log/superset/logship.logis empty, or shows onerefused: <code>line for each refusal.ls /var/lib/superset/logship/shows one.offsetfile per source.
- In Sentry, open the sandbox project → Logs, and search
cloud_workspace.id:<id>. Expect:- boot lines from the current
boot.starton, withlog.source=bootandbox.boot_started_atset (epoch ms). - host-service lines with their own ISO times.
- No
[host-service:vitals]lines, and nomemguard ranklines. [sandbox] start hook running (pid N) (command of L chars), and no line of the hook command anywhere.hook.exitedorhook.failedwithexit=andms=.
- boot lines from the current
- Check what does not arrive: no text from
start-hook.logorcheckout.log. - Check the volume after one awake hour: about 40–50 records for a quiet box. A
log shipper: cap of … reachedrecord means the cap needs a look. - Turn our organization's flag off, wake the box, write a line, then turn it on and wake again: that line must not arrive.
- Boot time: in
boot.log, compareboot.starttohost.readyon a wake with the flag on and a wake with it off. This was not measured before merge. - If logship.log shows
refused: 403, the sandbox project does not take logs. Turn the fleet switch off and settle it with M7.
Evidence
- Evidence page: https://app.superset.sh/page/m6-box-log-shipper-dk7mbs
- Design: https://app.superset.sh/page/admin-cloud-workspace-cost-and-health-design-8z3d4d
Checks
packages/sandbox:bun test(84 pass, including 15 inlogship.test.ts),bun run build,bun run typecheckpackages/trpc:bun test src/lib/sandbox(47 pass),bun run typecheckpackages/shared:bun test(1692 pass),bun run typecheckpackages/host-service:bun test src/runtime/sandbox-self-seed src/runtime/boot-stamps(34 pass),bun run typecheckbun run lint(the repo lint script, as CI runs it): cleanturbo typecheck --concurrency=1across the repo (CI's concurrency 3 runs out of memory on the workspace VM)- The
boot-twicejob on this PR has a new check: with the flag off, no shipper stamp, process or state.
Not verified: a real cloud box, and whether the sandbox Sentry project accepts log envelopes (see the evidence page).
Summary by CodeRabbit
- New Features
- Sandbox box logs can now be sent to Sentry when enabled for both the organization and the fleet.
- Log delivery excludes setup scripts and clone output, filters selected entries, redacts credential patterns, and limits the amount of data sent.
- Improvements
- Sandbox hook messages report outcomes without displaying the hook command text.
- Boot logs now distinguish hook outcomes, including successful starts, failures, and cases where no hook runs.
- Documentation
- Added guidance on box-log delivery and the information it may contain.