ci: nothing execs into k3s before it has written its kubeconfig

Merged
HA
15h ago
+4-3

Summary

Since #390, 3 of 49 E2E jobs timed out in "Finalize k3s" with the node never Ready, the latest on main in run 39179973449. Its k3s log, kept since #400, shows the cause. At 00:18:58 k3s logged "failed to move process 96 to cgroup /init" and "failed to enable controller ... device or resource busy". At 00:19:06 the kubelet exited on "cgroup [kubepods] has some missing controllers: cpu, cpuset, hugetlb, memory", and k3s died with it.

Changes

At startup k3s moves every process out of the container's root cgroup and then enables the controllers there. Our two pre-pull loops ran docker exec k3s crictl info the moment the container started (00:18:58.46 in that run), so an exec'd process could land in the root cgroup in between. Both loops now wait for /etc/rancher/k3s/k3s.yaml, which k3s writes seconds after that setup. cicd.md says why.

Type of Change

  • Bug fix (non-breaking change that fixes an issue)
  • New feature (non-breaking change that adds functionality)
  • Breaking change (fix or feature that would cause existing functionality to change)
  • Documentation update
  • Refactoring (no functional changes)
  • CI/CD or tooling changes

Test Plan

  • Backend unit tests pass (cd backend && uv run pytest tests/unit)
  • Frontend unit tests pass (cd frontend && npm run test)
  • Backend E2E tests pass (docker compose exec -T backend sh -c 'uv sync --group test --no-dev --frozen --no-install-project --no-build && pytest tests/e2e')
  • Frontend E2E tests pass (cd frontend && npm run test:e2e)
  • API spec and client regenerated if routes or schemas changed (./deploy.sh openapi, then ./deploy.sh types), never edited by hand
  • Manual testing performed

Validation: with the same k3s image locally, execs already in flight at startup gave "device or resource busy" in 5 of 6 trials. No exec gave none. Here the node still came up, since this Docker host apparently has the controllers enabled already; the runner evidently didn't.

Checklist

  • My code follows the project's style guidelines
  • I have run pre-commit run --all-files and all checks pass
  • I have added tests that prove my fix/feature works
  • I have updated documentation if needed
  • My changes don't introduce new warnings

Summary by cubic

Fixes E2E jobs timing out in "Finalize k3s" because docker exec into the k3s container raced k3s's startup cgroup setup.

The pre-pull loops ran docker exec k3s crictl info the moment the container started. At startup k3s moves every process out of the container's root cgroup before enabling controllers there, so an exec landing in between made that fail with "device or resource busy", the kubelet exited, and the node never became Ready.

  • Both pre-pull loops now wait for /etc/rancher/k3s/k3s.yaml before exec'ing and document why.