ENG-13123: Retry transient scaling conflicts during deploy

Merged
KA
15d ago
+1588-45

When an app is still scaling, reflex deploy aborts while updating instance bounds or submitting the deployment. Both steps now wait and retry confirmed scaling refusals with bounded progress messages, preserving unrelated errors and avoiding replay of writes whose outcome is unknown.

Fixes ENG-13123.

Server dependency: https://github.com/reflex-dev/flexgen/pull/6265 must land and deploy to preserve hostname reservations across late scaling refusals. This CLI accepts its dedicated app_scaling code and retains an exact legacy app_busy message fallback for older servers. The hostname-reservation review thread remains open until that dependency is deployed.

Changes

  • Match confirmed HTTP 409 scaling refusals at the exact configured endpoint, including backend URL prefixes. Other busy states fail immediately.
  • Wait 15 seconds between scaling retries: seven for bounds and eleven for submission. A single per-deployment budget spans SDK safe retries, allowing at most fourteen submission attempts by default. The 165-second submission sleep budget excludes request time and SDK backoff.
  • Reuse serialized submission content, request ID, and uploaded archives; never export or upload again for scaling retries. Preserve interruption and uncertain-write handling.
  • Keep bounds URL and retry policy in the hosting helper. Distinguish server errors/lost responses from successful responses that cannot be decoded, warning about bounds potentially left changed when deployment stops.
  • Preserve SDK HTTP-library selection through a documented default_transport() export. Raise the hosting CLI's SDK floor to 0.1.1.dev0; release materialization must publish/lift that dependency before the CLI release.
  • Preserve GCP VM-type forwarding and the region-only notice. Add documentation and changelog fragments for both affected packages.

Validation

  • Regression tests failed before the corresponding fixes.
  • uv run pytest tests/units --cov --no-cov-on-fail --cov-report=: 11,700 passed, 26 skipped; 79.52% coverage.
  • 1,539 focused hosting CLI/SDK tests passed.
  • Repository Ruff lint/format and uv run pyright reflex tests packages/reflex-hosting-cli/src packages/reflex-build-sdk/src: passed.
  • uv run python scripts/check_min_deps.py reflex-build-sdk reflex-hosting-cli: both passed.
  • Independent adversarial review: no actionable findings. No live deployment was run.