improvement(credentials): keep valid client-credential tokens through refresh outages and pace token retries by failure kind

Merged
WA
3d ago
+1856-233

Summary

Makes client-credential token requests more resilient, following the patterns used by MSAL, Entra's resilience guidance and botocore. The changes apply to the per-instance mint cache, the shared coordinated resolver (Shopify, Linear, Atlassian, Vanta) and Vanta itself.

  • Still-valid tokens keep working. Tokens are refreshed early, inside the 5-minute window before expiry. If that refresh fails with an outage or throttling, the cached or stored token is served until 30 s before it expires. That margin is checked again after the failed refresh, so a slow attempt cannot carry the token past it. A refusal of the credentials still fails at once.

  • Retry timing depends on the kind of failure (shared policy in lib/credentials/token-request-resilience.ts):

    • Refusal (bad or revoked credentials): replayed for 30 s, keyed by the secret's fingerprint.
    • Throttled (429, or 503 with Retry-After): the provider's Retry-After is honoured (seconds or HTTP-date), clamped to 1 s to 5 min. Without one, wait 1 min, doubling up to 5 min.
    • Outage (other 5xx, timeout, network error, malformed or unreadable response): 5 s, doubling to 30 s while it lasts. Before the backoff, one retry inside the mint after about 1 s, only if the failed attempt was quick.
    • Each kind keeps its own streak. A 429 after a run of outages starts its backoff at 1 min and does not inherit the outage count.

  • Retry-After is carried on errors from every client-credential minter. TokenServiceAccountValidationError gets an optional retryAfterMs, read by throttledRetryAfterMs (429/503 only). It is filled in by:

    • the shared helpers: throwForProviderResponse, exchangeClientCredentialsGrant, Shopify and Vanta
    • every bespoke minter: Box, Zoom, DocuSign, NetSuite, Help Scout, Ramp, Jama, Zendesk, Salesforce, Zoho Desk
    • the CrowdStrike, SailPoint and ZoomInfo client errors, through a new optional field

    A replayed failure carries an optional replayForMs, which the per-instance layer honours exactly.

  • Coordinated resolver:

    • The failure is remembered once for every worker in the existing row. Remembered failures and the still-valid fallback are served without taking the application lock.
    • A refusal whose code the row cannot store (the previous build reads it as provider_unavailable) gets an additive refusal: true flag, so a replay never serves the stale token.
    • A throttled failure gets an additive throttled: true flag, so the throttling streak continues across workers and stays separate from an outage streak.
    • Vanta asks for a token at most every 15 s while exchanges fail, because Vanta allows 5 tokens a minute and each new token revokes the previous one. That includes a 200 whose body cannot be read.

  • Single-flight is unchanged: coalesceLocally per instance, the advisory lock across workers.

  • The new fallback warning leaves out the credential id in selector privacy mode, like every other log in credential-service.ts.

Type of Change

  • Bug fix
  • Other: resilience improvement

Testing

  • Unit tests in credential-service-cancellation.test.ts, errors.test.ts and client-credentials-grant.test.ts:
    • fallback while the token is valid, error once it has run out (including after a slow failed refresh), and no fallback after a refusal
    • the single in-mint retry
    • each replay window, the Retry-After cap and the backoff floors
    • a 429 after two outages waits about a minute, not the outage streak's longer wait
  • A Retry-After test for each bespoke minter, in the minter test files (new helpscout.test.ts).
  • Real-Postgres integration tests in coordinated-client-token.integration.ts and vanta-token.integration.ts, run one file at a time:
    • fallback across workers
    • the retry while holding the lock, shared by waiting resolutions
    • serving without taking the lock
    • backoff growth and reset after recovery
    • a throttled answer after an outage backs off from 1 min, and throttling that continues backs off longer for every worker
    • sweep retention
    • a refusal the row cannot store stays a refusal on replay
    • Vanta's 15 s pacing, including an unreadable 200 body
  • Rolling-deploy fixtures pin both earlier failure decoders. A refusal memo is still exactly {code,status}. An outage memo adds only consecutiveFailures, a throttle memo adds consecutiveFailures and throttled, and an unstorable refusal adds only refusal; the old decoders ignore all three. The replay length rides on the existing expiresAt.
  • Each guard was reverted and its tests went red.
  • For reviewers: check the 5-minute Retry-After cap, and that the retry while holding the lock stays inside the 15 s lock_timeout (2 s budget + about 1.2 s pause + one 10 s attempt).
  • Follow-up, not in this PR: the Microsoft app (Entra client-credentials) token cache has its own LRU and still has no stale fallback or failure memo.

Checklist

  • Code follows project style guidelines
  • Self-reviewed my changes
  • Tests added/updated and passing
  • No new warnings introduced
  • I confirm that I have read and agree to the terms outlined in the Contributor License Agreement (CLA)