Respawn failed Granian workers in production

Merged
MA
1d ago
+82-8

Fixes FINDING-033 from the reflex 0.11.0a1 pre-release testing (artifacts on branch claude/vibrant-pasteur-3921g3, prerelease-testing/runs/2026-11-11-v0.11.0a1/redis_runtime/ and verify_rds/).

All Submissions:

  • Have you followed the guidelines stated in CONTRIBUTING.md?
  • Have you checked to ensure there aren't any other open Pull Requests for the desired changed?

Type of change

  • Bug fix (non-breaking change which fixes an issue)
  • This change requires a documentation update (included)

Changes To Core Features:

  • Have you added an explanation of what your changes do and why you'd like us to include them?
  • Have you written new tests for your core changes, as applicable?
  • Have you successfully ran tests with your changes locally?

Defect

run_granian_backend_prod (reflex/utils/exec.py) builds granian.server.Server(...) without respawn_failed_workers, which granian defaults to False: when any worker dies (e.g. the OOM killer, kill -9), granian's serve loop breaks and the whole production server stops, disconnecting every client on every other worker. GRANIAN_RESPAWN_FAILED_WORKERS is only read by granian's CLI, so reflex run --env prod users had no way to turn respawning on. Not a regression (0.10.1 behaves the same).

Fix

  • Prod passes respawn_failed_workers=True by default (supported since the current floor granian[reload] >=2.7.4; checked 2.7.4 and 2.8.4). Opt out with GRANIAN_RESPAWN_FAILED_WORKERS=false — granian's own variable name, read with os.getenv like the neighbouring GRANIAN_WORKERS / GRANIAN_LOG_LEVEL, parsed with click.BOOL as granian's own CLI parses it, so every value granian accepts works (true/false, on/off, t/f, 1/0, yes/no); an invalid value is a click usage error (exit 2). No new reflex-namespaced setting.
  • The prod options are built by a small _granian_prod_options helper, which the existing JSON-logs prod test now uses too.
  • Dev (run_granian_backend) and the uvicorn/gunicorn backend (gunicorn already replaces dead workers) are untouched. The prod frontend static server goes through the same function and gets respawn too.
  • Docs: docs/hosting/self-hosting.md describes respawning, the opt-out, and GRANIAN_LOG_LEVEL=info to see respawns (granian logs "Respawning worker" at INFO; by default only [ERROR] Unexpected exit from worker-1 is shown — granian exposes no respawn hook, so no extra reflex log line is added).

Crash loops still end: with a worker that dies at startup every time, reflex exits rc=1 with granian's "Worker crash loop detected, exiting" (1 worker: after 2 respawns, ~0.2 s; 2-8 workers: within ~5 s). A worker that dies more than ~5.5 s apart each time is respawned indefinitely, like gunicorn.

Test plan

  • tests/units/utils/test_exec.py: test_run_granian_backend_prod_respawns_failed_workers[None/true/on/false/off] (before: KeyError: 'respawn_failed_workers') and ..._rejects_invalid_respawn_setting (before: DID NOT RAISE click.BadParameter); 56 passed after.
  • Full unit suite: 15639 passed, 66 skipped, 81.42% (the Playwright browser-driver module excluded locally for lack of a Chromium binary). ruff, pyright: clean.
  • Manual end to end (real reflex run --env prod, Redis, 2 workers): kill -9 of a worker → server keeps serving, a new pid serves requests after ~3 s; with GRANIAN_RESPAWN_FAILED_WORKERS=false → "Reflex app stopped." as before. With GRANIAN_LOG_LEVEL=info: [ERROR] Unexpected exit from worker-1 / [INFO] Respawning worker-1 / [INFO] Spawning worker-1 with PID: … / [INFO] Started worker-1.

Notes

  • Default-behaviour change for deployments that relied on fail-fast behind a supervisor (k8s, systemd): opt out as above (news fragment and docs say so).
  • Not addressed (FINDING-032, separate follow-up): a tab whose worker was killed still gets a new token on reconnect, because the dead worker's socket record outlives it; with this PR the server stays up but that session is still lost.
  • A respawned worker re-runs _setup_event_processor (fresh token-manager id; with #7795 also a fresh state-manager id).

News fragment: news/+granian-respawn-failed-workers.bugfix.md.


Generated by Claude Code