fix(etl-uvicorn): declare a rate limit retryable and normalize plugin_error.error_reason

Merged
PA
2mo ago
+221-5

What & why

Problem: A consumer that trusts this library's retryable flag is told a provider rate limit is permanent, which is the one failure whose entire meaning is "try again later". plugin_error_of hard-coded retryable = false AND error_type = "configuration" for the whole legacy UserError family; RateLimitError carries status_code = 429 and inherited both, so a throttled request serialized as configuration / invalid_input / user / retryable = false -- terminal, and typed as caller misconfiguration rather than a dependency failure. The taxonomy this envelope implements says the opposite on both counts: a 429 from the caller's own provider is user-audience and retryable, and error.type, audience and retryable answer three independent questions, so retryability is orthogonal to audience rather than derivable from it. Separately, plugin_error.error_reason was assigned straight from the raised error's failure_category, so any plugin that set one put a SCREAMING_SNAKE token (AUTH_PERMISSION_DENIED) on a field the same taxonomy specifies as lower_snake_case. These are two distinct vocabularies and the platform already treats them as such elsewhere -- a sibling module maps each SCREAMING_SNAKE category onto its own distinct lower_snake reason -- and this library collapsed them into one. Both defects are this library mis-declaring the error contract it publishes to every plugin that imports it.

Change: RateLimitError now serializes error_type = "dependency", audience = "user", retryable = true and an error_reason defaulting to rate_limited; every other member of the family keeps configuration / invalid_input / user / retryable = false, because terminal is the safe default and the transient condition is the one that has to declare itself. error_reason is now normalized to a lower_snake_case token rather than copied verbatim, and the top-level failure_category response field still carries the plugin's original string unchanged.

Known gap left alone

For any error that is not a UserError, plugin_error_of returns None and emits no envelope at all, so audience is absent rather than merely wrong, and a plugin has no way to declare a non-user audience. That is deliberately untouched here. The repository's own test pins the current behaviour, and changing it would add a new plugin-facing contract surface -- a design decision about what a plugin may declare, not a bug fix. Calling it out so it is acknowledged rather than discovered.

Risk / rollback

Low. A pure mapping function with no state, no migration, and no configuration. Back it out by reverting the PR, or by pinning consumers to 0.1.0.

How it was verified

Ran the library's own suite (make check-ruff clean on the repo's pinned ruff; full pytest run green, six new tests added and no existing test modified or deleted). Wrote a probe that serializes the whole unstructured_ingest.error family through plugin_error_of and prints every field a consumer reads off the wire, ran it against the base revision and against this branch, and diffed the two -- output below. The consumer enumeration above was checked against each repository's origin/main, not a local checkout. Not verified: no deployed environment was exercised, because this package is the wrapper library a plugin imports rather than a deployed service, so a real local run of the wrapper plus its suite is the closest real environment. The retryability flip in particular cannot be proven end to end from a consumer's behaviour today, since nothing reads the field yet.

Proof

Repro (on the base revision), same probe both times:

error                   status  error_type      error_reason              audience    retryable
UserError               401     configuration   invalid_input             user        False
UserAuthError           401     configuration   invalid_input             user        False
RateLimitError          429     configuration   invalid_input             user        False
QuotaError              401     configuration   invalid_input             user        False
ProviderError           500     (no envelope emitted)
CategorizedUserError    401     configuration   AUTH_PERMISSION_DENIED    user        False

A 429 reached the wire terminal and typed as a misconfiguration, and a plugin-supplied AUTH_PERMISSION_DENIED landed verbatim on a lower_snake_case field.

Failing test before the fix:

$ pytest test/api/test_api.py -q -k "rate_limit or error_reason or non_rate_limited or failure_category"
>       assert plugin_error["retryable"] is True
E       assert False is True
>       assert body["plugin_error"]["error_reason"] == "auth_permission_denied"
E       AssertionError: assert 'AUTH_PERMISSION_DENIED' == 'auth_permission_denied'
>       assert body["plugin_error"]["error_reason"] == "auth_permission_denied"
E       AssertionError: assert 'Auth Permission-Denied ' == 'auth_permission_denied'
>       assert client.post("/invoke").json()["plugin_error"]["error_reason"] == "invalid_input"
E       AssertionError: assert '///' == 'invalid_input'
FAILED test/api/test_api.py::test_rate_limit_error_reaches_the_wire_as_retryable
FAILED test/api/test_api.py::test_precheck_rate_limit_error_is_retryable
FAILED test/api/test_api.py::test_error_reason_is_lower_snake_case_on_the_wire
FAILED test/api/test_api.py::test_error_reason_normalizes_a_non_snake_failure_category
FAILED test/api/test_api.py::test_unusable_failure_category_falls_back_to_the_default_reason
5 failed, 5 passed, 58 deselected

After the fix, same probe and same command. Only the two rows the change targets moved:

BEFORE (base)                                    AFTER (this branch)
UserError      configuration invalid_input F     UserError      configuration invalid_input          F
UserAuthError  configuration invalid_input F     UserAuthError  configuration invalid_input          F
RateLimitError configuration invalid_input F     RateLimitError dependency    rate_limited           T
QuotaError     configuration invalid_input F     QuotaError     configuration invalid_input          F
ProviderError  (no envelope emitted)             ProviderError  (no envelope emitted)
Categorized... configuration AUTH_PERMISSION...  Categorized... configuration auth_permission_denied F

The four rows that should not move are byte-identical, and ProviderError still emits no envelope -- the gap described above, reported and deliberately not fixed here.

Why normalize rather than lowercase: failure_category is arbitrary plugin-supplied text validated only as a string, so .lower() on "Auth Permission-Denied " yields a space rather than a snake_case token. Punctuation and whitespace runs collapse to single underscores, and a category with no usable characters ("///") falls back to the class default instead of putting garbage on the wire. Both cases are pinned by tests.

Dependencies / merge order

none