feat: limit how many executions a user can have in flight
Limits how many executions one user can have queued or running at once, and makes the queue's slots expire so a lost release can't hold one forever. Carried over from #113, rebuilt on the Redis execution queue main already has.
The API reserves a slot in exec_queue:user:{user_id} before it writes the execution, in one Lua call that drops expired entries, checks the count and adds the new id. At max_executions_per_user (a new admin setting next to max concurrent executions, default 100, 1 to 1000) it answers 429. Retries go through the same path. A failed insert or publish gives the slot back, and the orchestrator releases it when the execution reaches a terminal state, including when it gives up after three failed saga starts, but not when it requeues after one. A reservation lasts 24 hours, as long as a queued event, so a release that never happens frees itself.
Running slots move from the exec_queue:active set to an exec_queue:running sorted set scored by an expiry: the longest pod deadline plus 300 s. The scheduler drops expired entries before it counts, so a slot whose release was lost (a saga timeout, a dropped event) comes back on its own. That covers the queue-slot half of #297; the saga paths that skip the release are still open there. The old set is left untouched in Redis and nothing reads it anymore.
DomainExecutionCreate now carries the execution id so the reservation can use it before the insert. The spec and client are regenerated with deploy.sh openapi and types, and the admin settings page has the new field.
Tested locally: ruff, mypy strict, grimp, 502 unit and contract tests, svelte-check, eslint, prettier, 1139 vitest tests, mkdocs strict, and the Generated API check. The non-k8s e2e suite against real Mongo, Redis and Kafka passed 432, including new tests for the reservation and expiry Lua; 19 of the 20 failures need k3s, and the other is the DLQ test #311 fixes on main.
Summary by cubic
Limits how many executions a user can have queued or running at once, and makes queue slots expire so a lost release can't hold one forever.
- The API reserves a Redis slot before creating an execution and answers 429 once the user reaches the new
max_executions_per_useradmin setting (default 100, 1–1000); retries go through the same path, and failed inserts or publishes return the slot. - Running and user slots are sorted sets scored by their expiry; once the event is loaded, both take the execution's own pod deadline (timeout raised to the language's minimum plus 300 s) via the shared
PodRuntimeConfig.timeout_for, and they're pruned at schedule time. A lost release frees them on their own. - A per-user reservation lasts 24 hours; the orchestrator releases it when the execution reaches a terminal state, including after three failed saga starts, but not on a single retry.
DomainExecutionCreatenow carries the execution id so the reservation happens before the insert, and the OpenAPI spec, generated types, and admin settings page are updated.