Per-request Gateway API keys: survey, Temporal design, and pipelex-server handoff (docs only)

Draft
LC
2mo ago
+726-0

Summary

Docs-only PR under wip/api_keys/. No code changes. It captures the study and design for giving each hosted run its own Portkey gateway key, and the handoff spec for the pipelex-server half of that work.

  • sdk-per-request-key-survey.md — where the API key is bound today (TOML placeholder → secrets provider → InferenceBackend → cached SDK client) and, per SDK we ship, whether the key can be overridden on the request rather than the client. Verified by executing the pinned versions against a recording httpx.MockTransport.
  • gateway-per-request-key-under-temporal.md — the narrowed design: Gateway-only, holds under our Temporal plugin. Reference (never material) rides on JobMetadata; a GatewayKeyResolver seam in the gateway plugin, defaulting to today's behaviour; the AsyncOpenAI vs AsyncPortkey override mechanics, including the post(headers=…) trap on the Portkey client that would pass review while sending the boot key.
  • pipelex-server-per-request-gateway-key-handoff.md — spec for the hosted plane: the runtime contract, what to build where (platform stamps the reference on both runner hops, a new resolver plugin member, provisioning, config/IAM/deploy), tests, and the open decisions. Corrects two design assumptions after checking pipelex-server dev: the runner never sees X-Org-Id (the platform fronts every /v1/* route and dispatches itself), and per-user Portkey keys already exist and now carry paid top-up credit.
  • …handoff.html — a skimmable HTML version of the handoff that points at the .md for details.
  • wip/google-structured-config-dropped-and-vertex-token-stale.md — two unrelated inference-provider bugs noticed during the survey, filed for later; not designed or fixed here.

Test plan

  • Docs only; nothing to run. Review is a read of the design and the handoff decisions (§6 of the handoff).

🤖 Generated with Claude Code

https://claude.ai/code/session_01Q3napiozmUbwpQBF7U6r53

Summary by cubic

Documents the design for per-request Portkey Gateway keys under Temporal and a pipelex-server handoff; no code changes. Today all hosted runs share one Gateway key; the design moves to a per-request key resolved from a reference on JobMetadata to enable per-tenant attribution, budgets, and limits.

  • Adds sdk-per-request-key-survey.md: where keys bind today and per-SDK override seams. Key trap: AsyncPortkey.post(headers=…) does not override auth; use with_options(api_key=…). openai paths accept extra_headers.
  • Adds gateway-per-request-key-under-temporal.md: narrow design. Introduces a gateway_key_ref on JobMetadata and a GatewayKeyResolver seam (default falls back to the boot key). All six Gateway handles override per request; retries re-resolve on any worker. No changes to SdkClientRegistry, InferenceManager, or instructor.
  • Adds pipelex-server-per-request-gateway-key-handoff.{md,html}: hosted implementation. Platform stamps gateway_key_ref on run dispatch payloads and sets X-Gateway-Key-Ref on /build/*; runner threads it to JobMetadata. New resolver member installed in both api-hosted and worker reads the tenant’s key (DynamoDB), with a process TTL LRU cache. Config switch for missing-reference policy; IAM read grants (and optional KMS decrypt). Corrects two assumptions: the runner never sees X-Org-Id; per-user Portkey keys already exist and carry paid credit.
  • Adds wip/google-structured-config-dropped-and-vertex-token-stale.md: files two unrelated defects found during the survey (Gemini structured config dropped via instructor, Vertex token minted once and never refreshed).

Decisions for reviewers

  • Choose the v1 reference: user id (reuses existing per-user keys/credit) or org id (new per-org keys).
  • Confirm where plaintext lives (row vs envelope encryption now).
  • Set cache numbers: positive TTL (revocation SLA), negative TTL, entry cap.
  • Policy: fail-closed on missing reference in hosted; keep fallback in local/dev.
  • Optionally ship alongside the org-scoped storage change to avoid duplicate runner/worker rebuilds and pipelex bumps.