cmux-vm image: cmux-machine-health timer (DNS, disk, RAM) and session/user-manager guards

Merged
TE
12d ago
+1051-1

Leo, about huge-blue (2026-10-25): "Like we need to have good ass images, good ass ergonomics, good ass everything."

Follows #25128 (DNS, landed). Part 2 (acpmux units, peer links and expected peers) is a follow-up PR on top of this one, to keep each under about 800 lines.

What broke on huge-blue

  • One ssh login session (session-c516566.scope) ran a tree of 849 git processes that used all the RAM and the disk.
  • The global OOM killer then took leo's systemd user manager, and acpmux with it.
  • The disk reached 0 free, and nothing alerted or healed.

What this changes (images/cmux-vm/machine-health/cmux-machine-health)

Guards, written by cmux-machine-health install:

  • /etc/systemd/system/session-.scope.d/50-cmux-guard.conf: MemoryHigh=70%, MemoryMax=80%, MemorySwapMax=25%, TasksMax=8192. The prefix drop-in applies to logind's transient session-N.scope units. systemd searches session-.scope.d/ for any session-*.scope, and transient units load drop-ins too. The smoke proves both on systemd 255 with a systemd-run --scope --unit=session-cmux-smoke-N.scope stand-in, which is the same StartTransientUnit path logind uses.
  • user@.service.d/50-cmux-guard.conf: OOMScoreAdjust=-900, MemoryMin=64M. user-.slice.d and user.slice.d get MemoryMin so the protection reaches the manager, since memory.min needs its ancestors to have it too. user.slice also gets MemoryMax=90%, so all sessions together leave system services room.
  • /etc/systemd/user.conf.d/50-cmux-guard.conf: DefaultOOMScoreAdjust=200, so the manager's -900 never reaches the services it starts.
  • User managers already running when install runs re-execute (SIGRTMIN+25, the same re-exec a systemd upgrade does; every service keeps running). Only once a manager reports DefaultOOMScoreAdjust=200 does its own process get -900, so no service can inherit it. A manager that does not confirm is left alone and protected from its next start.

Health loop: cmux-machine-health.timer runs cmux-machine-health.service every 60 s (OnActiveSec/OnUnitActiveSec, so a resumed snapshot does not fire a backlog). It is a oneshot run as root with TimeoutStartSec=45; the pass stops itself at 40 s and its commands are bounded by what is left of that budget.

  • DNS: resolves github.com, plus the machine's MagicDNS name when it is logged in to a tailnet. On failure, at most once per 15 min, it restarts systemd-resolved, relinks /etc/resolv.conf to the stub if it is not one (only when the stub lists a nameserver), restarts tailscaled if it is installed, then re-checks for up to 10 s.
  • Disk: checks free space on / against a floor. It starts each configured hygiene command as its own transient unit (systemd-run --no-block, RuntimeMaxSec=900, idle IO), at most once per 15 min, then re-checks. Under the hard floor the problem is reported as CRITICAL.
    • The default hygiene command is the existing cmux-machine-hygiene --apply for each /home user; the bake now ships it as /usr/local/bin/cmux-machine-hygiene.
    • Default floors: max(min(10 GB, 20% of disk), 5% of disk), and a hard floor of min(3 GB, 10%). The brief's flat max(10 GB, 5%) would flag a healthy 16 GB cloud VM, which has about 10 GB free. On huge-blue's 256 GB disk the floor is 12.8 GB and the hard floor 3 GB.
  • RAM: a MemAvailable floor of min(2 GiB, 10% of RAM). It only reports; the cgroup limits above do the killing.
  • Writes /run/cmux/machine-health.json atomically (mode 0644) per the shared contract: schema, checkedAt, problems (most important first, at most 8) and fixed (the last 24 h, at most 20).
    • Rate-limit state is in /run/cmux/machine-health-state.json (tmpfs). A corrupt state file is ignored, not fatal.
    • Details are cleaned: no hint tails, /home/x becomes ~, and token= is redacted.
  • Accounts come from /etc/passwd only, so a slow NSS source never stalls a pass.
  • Config: /etc/cmux/machine-health.json overrides the defaults (dnsNames, tailnetName, diskPath, diskFloorBytes, diskHardFloorBytes, ramFloorBytes, hygieneCommands as argv lists). install --config FILE validates and installs it (dnsNames must be host names). install --no-start enables the timer without starting it; the bake uses that. An invalid file falls back to the defaults and is reported as a loop problem.

Stop condition and cleanup: a timer-driven oneshot with a 45 s timeout and a 40 s internal budget. No process stays resident. An flock in /run/cmux allows one pass at a time. Every fix is rate-limited (DNS and disk once per 15 min). Hygiene runs in its own transient units with RuntimeMaxSec=900 and --collect. Remove it with sudo cmux-machine-health uninstall, or systemctl disable --now cmux-machine-health.timer. Uninstall stops the timer and removes the script, units and drop-ins; it keeps the config and /run.

Image:

  • bake.ts installMachineHealth stages the script, runs its own install (the same entry point huge-blue uses), systemd-analyze verifys the units, runs one pass, and records the JSON in bake-*.json.
  • smoke.ts machineHealthChecks checks on a clone:
    • the timer is active
    • one pass exits 0 with result success
    • the JSON is valid with no problems
    • a session scope stand-in gets pids.max 8192 and memory.high/max at 70%/80% of RAM (within 2%)
    • user@UID.service shows OOMScoreAdjust=-900 and MemoryMin=64M
    • a transient service under the work user's manager (systemd-run --user -M USER@) gets OOM score 200, not the manager's -900

Evidence

  • test_cmux_machine_health.py (stdlib unittest, fake host) covers: a healthy pass in the contract shape, DNS heal order (resolved, relink, tailscaled), DNS and disk rate limits with since kept, the tailnet name only when logged in, hygiene as transient units with critical under the hard floor, floors scaling and config overrides, fixed kept 24 h and capped at 20, a broken check reported while the others still run, a slow machine still writing its report inside the budget, redaction, bad config, and install/uninstall writing and removing exactly the units.
  • web/tests/vm-image-cmux-vm-machine-health.test.ts runs that suite, bash -n on the bake and smoke commands, and the smoke parser on healthy and broken outputs. It was added to cloud-vm-image-lock.yml.
  • Fleet Linux run (cmux-ci, Ubuntu 24.04, systemd 255): the unit tests, plus a real install --no-start with the bake command, the arm step and the smoke command and its parser, on the runner. See the comment below for the run link and counts.
  • Bake and smoke on a real VM: dispatched once this lands, since the bake runs only on feat-cmux-next. The result goes in a comment here, with idle CPU-s/min before (0.6451, bake run 40082595774) and after.

Install on huge-blue, from a checkout pinned to this PR's merge commit:

sudo python3 images/cmux-vm/machine-health/cmux-machine-health install --config /path/to/huge-blue-machine-health.json

With huge-blue's config, e.g. {"hygieneCommands": [["systemctl", "start", "--no-block", "disk-hygiene-pressure.service"]]}. Running user managers are re-executed in place (services keep running) and protected once they confirm; install never restarts them.

Risk: medium-low. New drop-ins change the limits on every future ssh and login session, and install re-executes running user managers in place. A session over 80% of RAM is now killed inside its own scope instead of taking the machine down. Nothing changes for acpmux sessions or system services.

Agent session: Air-Blue

šŸ¤– Generated with Claude Code