ci: E2E waits until the Kafka consumer groups are stable

Merged
HA
17h ago
+10-1

Summary

Small bug found while checking main's CI after #396: run 39171533369 failed two execution tests on a 30 s timeout. The worker's consumer groups spent 38 s rebalancing, and "FastStream app started successfully!" only came at 22:20:51. The tests submitted around 22:20:50, and the first saga of the run started at 22:21:20. On a fresh Kafka the saga group starts at the newest message, so a request sent before it joins is never read.

Changes

After the backend health check, e2e-ready waits until k8s-worker, result-processor, saga-orchestrator, notification-service and sse-bridge-pool report Stable in kafka-consumer-groups --describe --state, up to 120 s each, and prints every group's state if one never gets there. cicd.md says why. Production keeps its committed offsets across restarts, so the app code stays as is.

Type of Change

  • Bug fix (non-breaking change that fixes an issue)
  • New feature (non-breaking change that adds functionality)
  • Breaking change (fix or feature that would cause existing functionality to change)
  • Documentation update
  • Refactoring (no functional changes)
  • CI/CD or tooling changes

Test Plan

  • Backend unit tests pass (cd backend && uv run pytest tests/unit)
  • Frontend unit tests pass (cd frontend && npm run test)
  • Backend E2E tests pass (docker compose exec -T backend sh -c 'uv sync --group test --no-dev --frozen --no-install-project --no-build && pytest tests/e2e')
  • Frontend E2E tests pass (cd frontend && npm run test:e2e)
  • API spec and client regenerated if routes or schemas changed (./deploy.sh openapi, then ./deploy.sh types), never edited by hand
  • Manual testing performed

Validation: the same loop against a local Kafka with a fresh worker saw k8s-worker Stable after 11 s, result-processor after 24 s and saga-orchestrator after 41 s. This PR's own E2E jobs run the new step.

Checklist

  • My code follows the project's style guidelines
  • I have run pre-commit run --all-files and all checks pass
  • I have added tests that prove my fix/feature works
  • I have updated documentation if needed
  • My changes don't introduce new warnings

Summary by cubic

Fixes flaky E2E runs where execution tests timed out because they submitted requests while Kafka consumer groups were still joining. On a fresh Kafka most groups start at the newest message, so a request sent before they join is never read.

e2e-ready now waits for k8s-worker, result-processor, saga-orchestrator, notification-service, and sse-bridge-pool to report Stable in kafka-consumer-groups --describe --state (up to 120 s each) and prints every group's state if one never gets there. docs/operations/cicd.md explains why this is needed.