feat: use flex polling for async inference

Merged
PJ
1mo ago
+4433-321

Summary

Default text calls now submit a background Responses request with the flex service tier and retrieve the result by response ID. This avoids creating an uploaded file and batch for each individual call. Chat Completions inputs and outputs are translated using official SDK types. Explicit 24-hour text requests and all embeddings retain the Batch API lifecycle.

Both SDKs limit simultaneous flex HTTP operations to 20 per client, releasing slots between polls. Polling uses jitter and bounded retries with exponential backoff for transient retrieval failures, without resubmitting accepted work. FlexPollingError exposes the accepted response ID and original error when polling stops. The concurrency limit and consecutive retry count are configurable with matching Python and TypeScript options.

Incomplete results preserve partial output and usage, with Chat finish reasons for token limits and content filtering. Cancellation preserves per-request credentials, routing options, and timeouts. Python streaming create accessors cannot bypass routing. The proxy rejects oversized uploads immediately, and chat translation preserves provider extension fields. Documentation explains polling, concurrency, and recovery behavior.

Validation

  • Python: 135 tests and ty check pass with the installed SDK and minimum supported OpenAI 2.0.0.
  • TypeScript: 46 tests, typecheck, and build pass with the locked SDK and minimum supported OpenAI 4.104.0.
  • Regression coverage includes retry recovery without resubmission, exhausted/permanent failures retaining IDs, retry-count reset, jitter/backoff and Retry-After, HTTP concurrency, cancelled waiters, partial output, and cancellation transport options.
  • CI exercises both supported SDK configurations for each language.

The HTTP concurrency limit applies per client and does not cap queued server jobs or inherited resource calls. Database capacity has not been load-tested as part of this SDK change.