fix: a Kafka outage no longer leaves finished pods behind or fails logins
Summary
Fixes #298
A failed Kafka publish set publish_failed on the stored event, and nothing ever read it. Since #428 the result is written before anything is published, so the execution no longer stays queued. But with Kafka down when a pod finishes, the result processor's error still escaped to the pod monitor. The monitor then skipped deleting the pod and freeing the slot. Login and register answered 500 too, although nothing consumes those events.
Changes
The flag and mark_publish_failed go. Prod has 0 flagged events among the 39 stored since 2026-10-12. Events sent after their work is done go through a new produce_or_log: it still raises a MongoDB error, since the stored event is the record, but only logs a Kafka one. That covers auth and settings events, pod progress, pod_created, and the result processor's terminal event and result_stored. A failed result write still publishes result_failed. A publish failure after the write doesn't, since the result was stored. Executions, cancels and replays keep raising; their callers handle it.
Type of Change
- Bug fix (non-breaking change that fixes an issue)
- New feature (non-breaking change that adds functionality)
- Breaking change (fix or feature that would cause existing functionality to change)
- Documentation update
- Refactoring (no functional changes)
- CI/CD or tooling changes
Test Plan
- Backend unit tests pass (
cd backend && uv run pytest tests/unit) - Frontend unit tests pass (
cd frontend && npm run test) - Backend E2E tests pass (
docker compose exec -T backend sh -c 'uv sync --group test --no-dev --frozen --no-install-project --no-build && pytest tests/e2e') - Frontend E2E tests pass (
cd frontend && npm run test:e2e) - API spec and client regenerated if routes or schemas changed (
./deploy.sh openapi, then./deploy.sh types), never edited by hand - Manual testing performed
Validation: 542 unit tests, ruff, mypy strict, the orphan check and mkdocs build --strict pass. The new outage test fails on main with KafkaError and passes here. Backend e2e is left to CI.
Checklist
- My code follows the project's style guidelines
- I have run
pre-commit run --all-filesand all checks pass - I have added tests that prove my fix/feature works
- I have updated documentation if needed
- My changes don't introduce new warnings
Summary by cubic
Fixes #298. A Kafka outage no longer leaves finished pods behind or fails logins, and the unread publish_failed flag goes.
When Kafka was down and a pod finished, the result processor's publish error escaped to the pod monitor, which then skipped deleting the pod and freeing its slot. Login and register answered 500 too, although nothing consumes those events. The publish_failed flag set on every failed publish was never read by anything.
- Adds
produce_or_log, which stores an event likeproducebut only logs a Kafka error; a MongoDB error still raises, since the stored event is the record. - Routes auth and settings events, pod progress,
pod_created, and the result processor's terminal event andresult_storedthrough it. - A failed result write still publishes
result_failed; a publish failure after the write doesn't, since the result was stored. - Executions, cancels, and replays keep raising; their callers handle the error.
- Removes
mark_publish_failedand thepublish_failedfields. Prod has 0 flagged events among the 39 stored since 2026-10-12.