refactor: the worker runs executions without a saga, and cancel stops the pod
Summary
The worker ran one job as four services that sent each other commands through Kafka inside one process, and their copies of an execution's state drifted apart. Cancelling a running execution only marked the saga and freed its slot (saga_orchestrator.py), so the pod ran on, the execution ended as completed and the next queued run started beside it.
Changes
A new execution scheduler keeps the Redis queue and calls the k8s worker directly. The pod monitor hands a finished pod to the result processor and frees its slot. The processor writes first and publishes only if the write applied, so a cancelled pod's late failure sends nothing. Cancel stores cancelled, then deletes the pod. The saga goes with its routes, admin page, 300 s timeout and the saga and pod command events. The worker's group keeps the name saga-orchestrator so its offsets carry over.
[!WARNING]
Stored events and DLQ rows of the removed types no longer load, and the event pages fail on them until they're gone. Right after the deploy, on prod MongoDB:
const removed = ["saga_started", "saga_completed", "saga_failed", "saga_cancelled", "saga_compensating", "saga_compensated", "create_pod_command", "delete_pod_command", "allocate_resources_command", "release_resources_command"];
db.events.deleteMany({ event_type: { $in: removed } });
db.dlq_messages.deleteMany({ "event.event_type": { $in: removed } });
db.sagas.drop();
Type of Change
- Bug fix (non-breaking change that fixes an issue)
- New feature (non-breaking change that adds functionality)
- Breaking change (fix or feature that would cause existing functionality to change)
- Documentation update
- Refactoring (no functional changes)
- CI/CD or tooling changes
Test Plan
- Backend unit tests pass (
cd backend && uv run pytest tests/unit) - Frontend unit tests pass (
cd frontend && npm run test) - Backend E2E tests pass (
docker compose exec -T backend sh -c 'uv sync --group test --no-dev --frozen --no-install-project --no-build && pytest tests/e2e') - Frontend E2E tests pass (
cd frontend && npm run test:e2e) - API spec and client regenerated if routes or schemas changed (
./deploy.sh openapi, then./deploy.sh types), never edited by hand - Manual testing performed
Validation: backend e2e on k3s with this branch's worker gives 463 passed, 1 skipped (main collects 494; the 30 saga tests went). The cancel test now checks the stored cancelled. CI's e2e-ready also waits for a consumer on execution_requested, since the worker's two subscribers join seconds apart. The unit, contract, Vitest, lint, type and docs gates pass.
Checklist
- My code follows the project's style guidelines
- I have run
pre-commit run --all-filesand all checks pass - I have added tests that prove my fix/feature works
- I have updated documentation if needed
- My changes don't introduce new warnings
Summary by cubic
The worker no longer runs executions through a saga. A new execution scheduler pulls from the Redis queue and tells the k8s worker to start a pod directly; the pod monitor hands finished pods to the result processor, which writes the result before publishing anything and then frees its slot. Before, the four services inside the worker passed commands to each other over Kafka and kept drifting copies of the same execution state.
Cancelling a run previously only marked the saga and freed its slot, so the pod ran to completion. Cancel now stores the execution as cancelled and deletes its pod, ending the run at once. The saga orchestrator, its API and admin pages, and the saga and pod command events are removed; the worker's group keeps the name saga-orchestrator so its Kafka offsets carry over.
Migration
- Stored events and DLQ rows of the removed saga types no longer load, and the event pages fail on them. After deploying, run on MongoDB:
const removed = ["saga_started", "saga_completed", "saga_failed", "saga_cancelled", "saga_compensating", "saga_compensated", "create_pod_command", "delete_pod_command", "allocate_resources_command", "release_resources_command"];
db.events.deleteMany({ event_type: { $in: removed } });
db.dlq_messages.deleteMany({ "event.event_type": { $in: removed } });
db.sagas.drop();
```<!-- End of auto-generated description by cubic. -->