Respawn failed Granian workers in production
Fixes FINDING-033 from the reflex 0.11.0a1 pre-release testing (artifacts on branch claude/vibrant-pasteur-3921g3, prerelease-testing/runs/2026-11-11-v0.11.0a1/redis_runtime/ and verify_rds/).
All Submissions:
- Have you followed the guidelines stated in CONTRIBUTING.md?
- Have you checked to ensure there aren't any other open Pull Requests for the desired changed?
Type of change
- Bug fix (non-breaking change which fixes an issue)
- This change requires a documentation update (included)
Changes To Core Features:
- Have you added an explanation of what your changes do and why you'd like us to include them?
- Have you written new tests for your core changes, as applicable?
- Have you successfully ran tests with your changes locally?
Defect
run_granian_backend_prod (reflex/utils/exec.py) builds granian.server.Server(...) without respawn_failed_workers, which granian defaults to False: when any worker dies (e.g. the OOM killer, kill -9), granian's serve loop breaks and the whole production server stops, disconnecting every client on every other worker. GRANIAN_RESPAWN_FAILED_WORKERS is only read by granian's CLI, so reflex run --env prod users had no way to turn respawning on. Not a regression (0.10.1 behaves the same).
Fix
- Prod passes
respawn_failed_workers=Trueby default (supported since the current floorgranian[reload] >=2.7.4; checked 2.7.4 and 2.8.4). Opt out withGRANIAN_RESPAWN_FAILED_WORKERS=false— granian's own variable name, read withos.getenvlike the neighbouringGRANIAN_WORKERS/GRANIAN_LOG_LEVEL, parsed withclick.BOOLas granian's own CLI parses it, so every value granian accepts works (true/false,on/off,t/f,1/0,yes/no); an invalid value is a click usage error (exit 2). No new reflex-namespaced setting. - The prod options are built by a small
_granian_prod_optionshelper, which the existing JSON-logs prod test now uses too. - Dev (
run_granian_backend) and the uvicorn/gunicorn backend (gunicorn already replaces dead workers) are untouched. The prod frontend static server goes through the same function and gets respawn too. - Docs:
docs/hosting/self-hosting.mddescribes respawning, the opt-out, andGRANIAN_LOG_LEVEL=infoto see respawns (granian logs "Respawning worker" at INFO; by default only[ERROR] Unexpected exit from worker-1is shown — granian exposes no respawn hook, so no extra reflex log line is added).
Crash loops still end: with a worker that dies at startup every time, reflex exits rc=1 with granian's "Worker crash loop detected, exiting" (1 worker: after 2 respawns, ~0.2 s; 2-8 workers: within ~5 s). A worker that dies more than ~5.5 s apart each time is respawned indefinitely, like gunicorn.
Test plan
tests/units/utils/test_exec.py:test_run_granian_backend_prod_respawns_failed_workers[None/true/on/false/off](before:KeyError: 'respawn_failed_workers') and..._rejects_invalid_respawn_setting(before:DID NOT RAISE click.BadParameter); 56 passed after.- Full unit suite: 15639 passed, 66 skipped, 81.42% (the Playwright browser-driver module excluded locally for lack of a Chromium binary).
ruff,pyright: clean. - Manual end to end (real
reflex run --env prod, Redis, 2 workers):kill -9of a worker → server keeps serving, a new pid serves requests after ~3 s; withGRANIAN_RESPAWN_FAILED_WORKERS=false→ "Reflex app stopped." as before. WithGRANIAN_LOG_LEVEL=info:[ERROR] Unexpected exit from worker-1/[INFO] Respawning worker-1/[INFO] Spawning worker-1 with PID: …/[INFO] Started worker-1.
Notes
- Default-behaviour change for deployments that relied on fail-fast behind a supervisor (k8s, systemd): opt out as above (news fragment and docs say so).
- Not addressed (FINDING-032, separate follow-up): a tab whose worker was killed still gets a new token on reconnect, because the dead worker's socket record outlives it; with this PR the server stays up but that session is still lost.
- A respawned worker re-runs
_setup_event_processor(fresh token-manager id; with #7795 also a fresh state-manager id).
News fragment: news/+granian-respawn-failed-workers.bugfix.md.
Generated by Claude Code