cmux-vm image: keep the resolver on systemd-resolved and check DNS survives a resolved/tailscaled restart

Merged
TE
12d ago
+433-2

Leo, about huge-blue (2026-10-25): "Like we need to have good ass images, good ass ergonomics, good ass everything."

What broke

On huge-blue, Tailscale ran in "direct" mode and wrote /etc/resolv.conf itself with "no upstream resolvers set", so nothing resolved. Someone fixed it by hand: /etc/resolv.conf now links to systemd-resolved's stub, and Tailscale is in resolved mode. Tailscale picks resolved mode only when resolv.conf is the resolved stub and resolved is running, so the link is what makes it safe. Persistent cloud machines need that guarantee too.

What this changes

  • images/cmux-vm/machine-health/cmux-dns-check: a standalone bash check that changes nothing except two restarts. It is baked as /usr/local/sbin/cmux-dns-check and runs as-is on huge-blue. It checks:

    • /etc/resolv.conf is a link to /run/systemd/resolve/stub-resolv.conf
    • systemd-resolved is active
    • Tailscale's DNS mode in the current tailscaled run's journal (dns: [... ret=MODE]); direct fails
    • a public name resolves (default github.com)
    • the machine's own MagicDNS name resolves when tailscale status --json shows BackendState Running and MagicDNS on; otherwise the check is skipped with the reason printed

    --restart first restarts systemd-resolved, then tailscaled when it is installed.

  • Bake (dns.ts resolverBakeCommand): keeps systemd-resolved enabled and active, and records whether resolv.conf already linked to the stub. If it did not, the bake makes the link, but only when the stub lists a nameserver. It also records the provider's resolved.conf.d entries without editing them (60-freestyle-vms.conf is untouched), then runs the check. The result is in bake-*.json as resolver.

  • Smoke: a new dns-resolved-restart check on a clone runs cmux-dns-check --restart after the automation checks.

  • Tailscale is not in the image (no package in the lock, the base dpkg list or the file manifest), and nothing in the image forces direct mode. On machines that do install Tailscale, the check reads the mode tailscaled actually picked.

Evidence

  • web/tests/vm-image-cmux-vm-dns.test.ts runs the script against stubbed systemctl, journalctl, getent and tailscale. Cases: a healthy machine, --restart order (resolved, then tailscaled), the huge-blue breakage (plain resolv.conf, ret=direct, nothing resolving: exit 1 with each FAIL), logged-out and MagicDNS-off tailnets (skip with the reason), and bad usage. It was added to cloud-vm-image-lock.yml.
  • Fleet Linux run (cmux-ci, run 40234697671): bun test of the dns, commands, lock and bake-workflow suites gave 57 pass, 0 fail. tsgo --noEmit and eslint on the changed files were both clean.
  • Bake and smoke on a real VM: cloud-vm-image-bake.yml runs only on feat-cmux-next, so it is dispatched after this lands. The result goes in a comment here.

Risk: low. One new image file, one bake step that only checks (and links only when the base has no link and resolved is serving), and one smoke check. No timer, loop or resident process.

Run on huge-blue: sudo bash images/cmux-vm/machine-health/cmux-dns-check --restart from a checkout at this commit.

Agent session: Air-Blue

šŸ¤– Generated with Claude Code


View with [code]smith Autofix with [code]smith
Need help on this PR? Tag @codesmith-bot with what you need.