test: e2e tests share one MongoDB, Redis and Kafka with the worker without racing each other
Third in the stack after #377 and #378. The backend e2e suite runs on pytest-xdist against one MongoDB, one Redis, one Kafka and the real worker, and several tests raced each other or the worker through that shared state.
Readiness waits subscribed to Redis pub/sub after the request that triggers the event, so a fast event was simply lost and the wait timed out. wait_for_event(execution_id, *types) polls the events collection instead. The producer writes there before it publishes, so an event that already happened still counts. The notification helper waits for its own notification, by the exec: tag, to reach DELIVERED, since delivery publishes to SSE before it updates the status.
test_settings gave each xdist process its own Redis database, but the worker always used DB 0. So per-user execution slots the API reserved on any other database were never released, and tests leaned on the split for isolation without anyone noticing. Everyone now shares the config's database. Tests that relied on the split use uuid usernames, keys and Redis prefixes, and the rate-limit tests build their own service instead of rewriting the prefix of the app-wide one and monkeypatching the shared client.
Sharing Redis exposed a lost update in the app. Every user's rate-limit override lived inside one config blob, and update_user_rate_limit read it, changed one user and wrote it all back, so two admins updating different users dropped one change. Each override is now its own field of a Redis hash, written with HSET. Reads merge the hash over the blob, so overrides already stored the old way still apply.
Other shared state:
- System settings: settings tests change only
metrics_retention_days,log_level,enable_tracingandsampling_rate, which nothing reads at runtime (#380 proposes removing them), starting from the current settings. The partial-update test is gone: the PUT replaces the document, so a partial body wrote a 300s timeout, 2000m CPU and 512Mi memory for every concurrent test. - Replays: the start, pause and resume tests replayed the whole events collection into Kafka and never stopped. They now use the TEST target, which publishes nothing, from a fixture that cancels the session.
- DLQ: the DLQ test left a real row for
saga-orchestratorthat the worker republished 30s later as a phantom execution. It now checks the stored row and the event handed to a mocked broker, targets a group nobody runs, and its fixture deletes the row. - Executions: tests that need an execution row but not a pod use
store_executionor the repository, so they no longer use up slots under the global concurrency limit. - Login lockout: failed-login tests use uuid usernames, since lockout counts are per username.
Cleanup: services and repositories come in as fixtures (execution_service, execution_repository, ...) instead of scope.get calls in each test. The 566 @pytest.mark.asyncio decorators are gone, since asyncio_mode = "auto" covers them, and so are the infra markers nothing selects on. xdist_group is left only on the two settings files, where it protects the one document. Python 3.11 is no longer hardcoded: requests use the registry's default, and CI pre-pulls only that image. Several list tests now assert exact contents for their fresh user instead of looping over a possibly empty list.
Tested on a local k3s in a container with the real worker: two full e2e runs on fresh infra, 488 passed and 1 skipped each. Ruff, mypy, the orphan check and unit tests pass. AGENTS.md's testing section describes the new isolation rules.