feat(python-client): retry a GET on 429 and 502-504 with exponential backoff

Merged
BU
8h ago
+114-3

What does this PR do?

A script that reads many variables or resources at once can get a 429 or a 502/503/504 back from a busy server. The Python client raised on the first one, so the script failed even though the same read would have worked a second later.

Windmill.get now retries a GET that comes back 429, 502, 503 or 504, up to 3 times, 1s, 2s and 4s apart. When the response carries Retry-After (delay-seconds or an HTTP-date), it waits that long instead; an unparseable value falls back to the backoff delay. One GET spends at most 15s sleeping in total: a wait that would exceed what is left of that budget ends the retries. After the last attempt it returns the last response, so the caller sees the same error as before. Every GET the client sends goes through Windmill.get (get_variable, get_resource, job status and results, ...), so they all get it. POSTs and every other status are not retried: a POST may not be safe to send twice, and a 4xx won't change on a retry.

Two new keyword arguments on Windmill.get:

  • retry_transient=False opts a GET out of retries. The Slack approval request uses it, since repeating a 502/504 there could post a second approval message.
  • retry_budget lowers the 15s budget. wait_job passes the time left before its own timeout, so retries cannot push it past that timeout.

The change is in python-client/wmill/wmill/client.py (+58/-3), plus a test file tests/test_get_retry.py. No new dependency. The TypeScript client is not touched.

Reproduced on main at a2c04c2aa8.

flowchart TD
    A[Script calls wmill.get_variable] --> B[Server answers 429 or 503]
    B -->|Before| C[The call raises at once]
    B -->|After| D[Wait 1s, 2s, 4s or Retry-After, then try again]
    D -->|Server recovers| E[The call returns the value]
    D -->|Still failing after 3 retries| F[The call raises the same error as before]

Narrated recording, before then after (sound on):

https://github.com/user-attachments/assets/61952a57-23b6-4744-964a-2fd09ab13311

The recording is a terminal running a small script that calls wmill.get_variable against a local stub of the variables endpoint. The stub logs each request it gets and the status it sends back. The first part imports the client from main, the rest from this branch. Nothing is edited.

Edge cases and unchanged behaviour checked

All checked in the recording, on this branch:

  • A 503, then a 503, then a 200: the read succeeds after waits of 1s and 2s.
  • A 429 with Retry-After: 3, then a 200: the client waits 3s and succeeds.
  • A server that stays at 503: 4 requests, 1s, 2s and 4s apart, then the original 503 error.
  • A 404 is not retried: one request, same error as before.
  • Unchanged: a normal 200 read is a single request.

Checks run

  • pytest tests/ in python-client/wmill (Python 3.14, as sdk-tests.yml runs it): 141 passed.
  • The new test tests/test_get_retry.py fails on main and passes on this branch. It also covers the 15s budget, an HTTP-date and an unparseable Retry-After, and retry_transient=False.

Related issue

Part of #2332. This covers the Python client's reads. The TypeScript client has no retry yet.

AI tools (Claude Code) assisted with this change; the behaviour was verified as shown.


Summary by cubic

Windmill.get now retries GETs answered with 429, 502, 503, or 504 so scripts reading many variables or resources no longer fail on the first transient server error.

Up to 3 retries run 1s, 2s, and 4s apart, or wait the Retry-After the response carries in seconds or HTTP-date form — falling back to the backoff delay when unparseable — with the total sleep bounded to 15s per GET. Callers whose GET has a side effect can opt out via retry_transient=False; the slack approval request does, since repeating its 502/504 would post a second approval. The job-result polling loop now passes a retry_budget matching its remaining timeout. After the last attempt the last response is returned, so callers get the same error as before. POSTs and other statuses are never retried.

Part of #2332.