fix(k3s): isolate containerd on dedicated SSD

Merged
SH
3mo ago
+271-17

Summary

  • mount Kyber containerd from the ext4 filesystem labeled k3s-containerd
  • make K3s require the mount so a missing SSD fails closed instead of consuming the root disk
  • add an explicitly destructive, guarded disk-preparation command for clean installs
  • order Home Manager activation so the mount is ready before K3s config can restart the service
  • document bootstrap, verification, and PVC/storage boundaries

Verification

  • make shell-test (1,676 ShellSpec examples and 420 Fish tests)
  • shellspec spec/k3s_service_activate_spec.sh spec/activate_k3s_spec.sh spec/coverage_spec.sh (131 examples)
  • shellcheck home-manager/services/k3s/activate.sh named-hosts/kyber/prepare-containerd-disk.sh
  • make nix-format-check
  • make build
  • Kyber activation DAG evaluation: setupK3s after writeBoundary, k3s-server after setupK3s

No disk was formatted, no Home Manager switch was run, and no live Kyber state was changed.


Summary by cubic

Isolated containerd for k3s on a dedicated ext4 SSD labeled k3s-containerd, mounted at /var/lib/rancher/k3s/agent/containerd. k3s now hard-requires this mount to avoid using the root disk and to reduce I/O contention.

  • New Features

    • Added a systemd mount unit resolving /dev/disk/by-label/k3s-containerd to the containerd path (no nofail), and updated k3s.service to Require and After the mount.
    • Activation installs/updates the mount unit, refuses risky states (k3s running or non-empty dir), enables the mount first, then k3s; k3s-server runs after setupK3s.
    • Hardened named-hosts/kyber/prepare-containerd-disk.sh: explicit --confirm-wipe, refuses non-block/mounted devices and active k3s, creates ext4 with the label, only reuses the label when it’s the same device and ext4, and verifies the mounted device matches. Docs and tests updated.
  • Migration

    • New or wiped host: sudo systemctl stop k3s, run ./named-hosts/kyber/prepare-containerd-disk.sh /dev/<device> --confirm-wipe, then make switch.
    • Verify: findmnt /var/lib/rancher/k3s/agent/containerd, systemctl is-enabled var-lib-rancher-k3s-agent-containerd.mount, systemctl show k3s -p Requires -p After, then restart k3s.