refactor: one worker process runs the consumers, the pod watch and the scheduled jobs
Six worker containers each loaded the whole app to run one or two Kafka subscribers or a timer. On prod they sat at 146 to 150 MiB of their 160 MiB limits at under 3% CPU, about 890 MiB together.
python -m app.worker (backend/app/worker.py) now runs all of it in one container: the k8s worker, result processor and saga routers, the pod watch, and the saga queue, saga timeout and DLQ retry jobs. Each consumer keeps its group id (k8s-worker, result-processor, saga-orchestrator), so Kafka delivers exactly what it did before and the committed offsets carry over. The service code doesn't change, only the wiring: one container factory instead of six, one config.worker.toml instead of six override files, and no backend/workers/ package. app.worker joins app.main as an entry point in the orphan check.
The event replay worker is gone rather than merged. Its cleanup job ran against an empty session map, since replays live in the API, so all it did was delete finished replay_sessions older than 48 hours. A TTL index on completed_at does that now. On first boot MongoDB will drop every finished session that ended more than 48 hours ago.
The worker has to stay at one replica, as the pod monitor did, since the pod watch and the jobs would run once per replica. Compose gives it 256m; it idles near 150 MiB.
Tested on a local k3s in a container, with Kueue, Mongo, Redis and Kafka on fresh state for each run: the backend e2e suite gives 491 passed, 1 skipped for this branch with the merged worker, and the same for main with its six workers. The merged worker used 146 to 150 MiB during the run, against 145 MiB for each of main's six. Ruff, mypy, the orphan check, unit tests, the Grafana contract and the strict docs build all pass.
Deploy note: the nightly release runs docker compose up --remove-orphans, which removes the six old containers.
Docs follow the change, including the system diagram (one worker box, replay edges moved to the backend) and the event replay page, now at components/event-replay.md.
Summary by cubic
Replaces the six worker containers with one worker process that runs the k8s worker, result processor and saga Kafka routers, the pod watch, and the saga queue, saga timeout and DLQ retry jobs. Each consumer keeps its consumer group, so Kafka delivery and committed offsets don't change; the six containers used about 890 MiB together, while the single process idles near 150 MiB.
Migration
- The worker must run at one replica, since the pod watch and scheduled jobs would run once per replica.
- The event replay worker is removed; a MongoDB TTL index on
completed_atnow drops finished replay sessions 48 hours after they end. - On the first boot after this change, MongoDB deletes every finished session older than 48 hours.
- The nightly release runs
docker compose up --remove-orphans, which removes the six old containers.