devbox image: install cmux-machine-health, parked into the snapshot and started by the clone's housekeeping re-arm

Merged
TE
12d ago
+338-21

Follows #25139 (merged into feat-cmux-next as bca1c4c5509).

Leo's words:

"Like we need to have good ass images, good ass ergonomics, good ass everything."

"these are honestly old as shit. we'd better just supersede them if we have better stuff now that we know more."

Supersedes #18670 (prior art by @austinywang)

What changes

The main devbox image now installs cmux-machine-health (from #25139) the same way the cmux VM image does. It reuses the helpers in web/scripts/cmux-vm-image/machine-health.ts by importing them, with no copy.

  • Bake (build-devbox-freestyle.ts): after cmux-tui-daemon-park, it writes the script and cmux-machine-hygiene with vm.fs.writeFile, then runs machineHealthBakeCommand(). That is install --no-start, systemd-analyze verify, and one run --check-only pass, so no fix and no disk hygiene runs during the bake. A new machine-health-parked step runs after clean, right before settle-before-snapshot. It deletes the bake pass's files and checks three things: the timer is enabled, the timer is not active, and the boot script lists it.
  • The devbox does not start the timer before its snapshot, unlike the cmux VM image. A devbox clone resumes with its clocks jumped by the snapshot's age. A timer that was running at the snapshot would fire in the clone's first second, alongside the daemon start, which is the reason cmux-devbox-boot already parks its HOUSEKEEPING_TIMERS. So cmux-machine-health.timer joins that list. The parked branch stops it, and a clone starts it HOUSEKEEPING_DELAY (10 min) after it binds. A cold boot starts it through timers.target, because it is enabled.
  • Park (devboxParkDaemonCommand): a size-derive clone parks again before its own snapshot, and by then its re-arm may have started the timer, a pass and disk hygiene units. Stopping a timer doesn't stop the units it started. So park first stops the re-arm, then the timer, then the pass and the cmux-machine-health-hygiene-* units (SIGKILL after 30 s; a pass writes its files atomically and hygiene renames before it deletes). It then proves none of them is active, and only then deletes /run/cmux/machine-health*.json.
  • Verify (verify-devbox-image.ts): right after resume, the clone must be parked: the health timer inactive and no health file. The re-arm must be pending as the boot script schedules it: OnActiveUSec equal to the boot script's HOUSEKEEPING_DELAY, target cmux-housekeeping-rearm.service, and a command that starts cmux-machine-health.timer. After every other check (the re-arm also starts the other housekeeping timers, which the quiet-resume check must not see), it starts cmux-housekeeping-rearm.service by hand, never the health timer, and requires the health timer to come up through it. If verification ran past the 10 min and the re-arm already fired, that counts too. Then it runs machineHealthSmokeCommand(cmux) / machineHealthSmokeProblems(). That smoke covers a timer-started pass, one pass with no problems, the session-scope and user.slice limits, and the work user's manager OOM score. Any problem fails verification.
  • Hygiene fit: by default the loop runs cmux-machine-hygiene --apply for each /home/NAME account with uid 1000 to 59999. The devbox work user is cmux, uid 1000, home /home/cmux, so the default covers it. The cmux-tui state reservation is a loop mount under that home, and hygiene refuses to touch mount points.
  • Source digest schema 3: devboxSourceManifest() now hashes DEVBOX_BAKED_PAYLOADS (the health script, cmux-machine-hygiene and cmux-vm-image/machine-health.ts), so a change to one of them alone asks for a rebake. Entries keep the schema they were recorded with. upgradeDevboxSourceRecords stops at schema 2 (DEVBOX_SOURCE_PROVABLE_SCHEMA), because git can't prove the payloads of an older bake.
  • Drift excuse: DEVBOX_AWAITING_REBAKE.current (schema 2) moves to the new desktop source digest, because the bake script and boot script changed. The defaults still run the old sources until the lane's next bake.
  • Tests (vm-devbox-image.test.ts):
    • Install and parked step: the install comes after the park; the parked step comes after clean and is the last step before settle-before-snapshot. The install is a check-only pass, with no systemctl start cmux-machine-health.service or .timer anywhere in the bake. The timer is in HOUSEKEEPING_TIMERS.
    • Park: it stops the units in order, checks, then deletes. A fake systemctl shows the stop passes once nothing is active and fails while a hygiene unit is still running.
    • Re-arm parsers: a wrong delay, a wrong target, a command that doesn't start the health timer, or a timer the re-arm didn't activate each fails verification.
    • Digest: changing any payload changes the schema-3 digest and leaves schema 2 alone.
    • Also: the uid and home fit the hygiene default, and bash -n passes on every command.

No workflow change was needed. cloud-vm-image-contract.yml already filters on build-devbox-freestyle.ts, verify-devbox-image.ts, devbox-image-common.ts, web/services/vms/images/devbox/** and web/tests/vm-image-*.test.ts.

Stop condition and cleanup

The loop is a timer-driven oneshot with TimeoutStartSec=45 and a 40 s pass budget, so there is no resident process. Every fix is rate limited: the DNS and disk fixes each run at most once per 15 min. sudo cmux-machine-health uninstall removes the timer, the service, the script and the guard drop-ins. On a devbox the timer also stops whenever the machine is parked for a snapshot.

Evidence

The fleet run (cmux-ci test, Linux, 32 vCPU) passed at 512c6eb5ca1: https://github.com/manaflow-ai/cmux/actions/runs/40266691666

  • bun test tests/vm-devbox-image.test.ts tests/vm-image-cmux-vm-machine-health.test.ts tests/vm-image-manifest.test.ts tests/vm-devbox-identity.test.ts: 101 pass, 0 fail
  • bun run devbox:manifest:check: ok, 12 validated defaults; sources awaiting their rebake: desktop c7f978ae7123ā€¦
  • bun run typecheck: clean
  • eslint on the 5 changed TS files: 0 errors (3 warnings, none on changed lines)

I have not run a real bake or verify. The lane dispatches those.

Risk: medium. This changes the bake, the boot supervisor's housekeeping list and the verifier for every new devbox image. Running machines change only after a rebake. If the bake's check-only pass or the parked check fails, the bake aborts before any snapshot exists. The new guard drop-ins put a ceiling on login sessions (MemoryMax 80%, TasksMax 8192) and on user.slice (90% of RAM). The verify smoke checks those limits on a clone.

Agent session: Air-Blue

šŸ¤– Generated with Claude Code