ci: wait up to 300 s for Kueue's controller and print its log on failure

Merged
HA
5h ago
+4-2

Summary

Main's run 39177020215 failed before any test: Kueue's controller started at once from the pre-pulled image, then answered every readiness probe with 404 for 110 s, and the 120 s wait gave up. This happens on a normal start, just a slow one.

Changes

Kueue v0.20.1 first runs a cert bootstrap manager on the health probe port with only a healthz check (BootstrapCerts), so /readyz is a 404 until bootstrap ends. Bootstrap waits for the kubelet to mount the new cert secret, and cert-controller v0.16.0 polls for that file with a doubling backoff from 1 s, so a slow mount pushes readiness past two minutes. A local controller took 1 min 44 s (20:33:30 "Waiting for certificate generation", 20:35:14 "Certificate bootstrap complete"). The wait is now 300 s, and on failure the step also prints the controller's last 300 log lines. cicd.md explains both.

Type of Change

  • Bug fix (non-breaking change that fixes an issue)
  • New feature (non-breaking change that adds functionality)
  • Breaking change (fix or feature that would cause existing functionality to change)
  • Documentation update
  • Refactoring (no functional changes)
  • CI/CD or tooling changes

Test Plan

  • Backend unit tests pass (cd backend && uv run pytest tests/unit)
  • Frontend unit tests pass (cd frontend && npm run test)
  • Backend E2E tests pass (docker compose exec -T backend sh -c 'uv sync --group test --no-dev --frozen --no-install-project --no-build && pytest tests/e2e')
  • Frontend E2E tests pass (cd frontend && npm run test:e2e)
  • API spec and client regenerated if routes or schemas changed (./deploy.sh openapi, then ./deploy.sh types), never edited by hand
  • Manual testing performed

Validation: the log command ran against a local Kueue controller (300 lines). This PR's own E2E run uses the new step.

Checklist

  • My code follows the project's style guidelines
  • I have run pre-commit run --all-files and all checks pass
  • I have added tests that prove my fix/feature works
  • I have updated documentation if needed
  • My changes don't introduce new warnings

Summary by cubic

Fixes CI failures where Kueue's controller never became ready within the 120-second wait, aborting E2E runs before any test started.

Kueue v0.20.1 boots a cert bootstrap manager that serves only /healthz until the new certificate secret is mounted, so /readyz returns 404 and the readiness probe fails. The bootstrap polls for the mounted file with exponential backoff, and a slow kubelet mount can push readiness past two minutes—a local controller took 1 min 44 s.

  • Increases the readiness wait from 120 s to 300 s.
  • Prints the controller's last 300 log lines when the wait fails, so failures are diagnosable without rerunning.
  • Updates docs/operations/cicd.md to explain both changes.