coderouter: refresh credentials rejected by usage reads, report revoked sign-ins as tenant faults

Merged
LA
15d ago
+532-103

Summary

Sentry CODEROUTER-WEB-18 (coderouter.provider_usage, upstream 401) grew to 172 events on 2026-10-07. Axiom shows one team's account-list polls hitting chatgpt.com/backend-api/wham/usage with about nine Codex access tokens that the provider had revoked before their recorded expiry. The usage read never refreshed a rejected token and never marked the account, so every dashboard or CLI poll re-read the dead credentials and filed one operator error per account. When a model request later reached the same account, /v1/responses forced a refresh, OpenAI answered 401 on /oauth/token, and the account went broken. That path reported the same failure twice: CODEROUTER-WEB-T from the refresher and CODEROUTER-WEB-V from the proxy, both at Sentry error.

This change makes the usage read on GET /api/coderouter/accounts follow the same recovery as the model routes. A 401 now forces one refresh under the existing lease and then retries the read. A rotated token returns usage normally. A revoked sign-in comes back as state: "broken", so the dashboard shows "Sign-in broken" now, and later polls skip that account. A busy lease defers the account to the next poll without a report.

A terminal refresh failure means the team's sign-in was revoked, so the refresher now reports it with fault: "tenant". Sentry and PostHog record it at warning with a fault:tenant tag. The proxy no longer reports a second time for CodeRouterCredentialBroken. Non-terminal refresh failures, usage 5xx, and transport errors still report as before.

Testing

tests/coderouter-usage-rejected-credential.test.ts is new. It failed on the first commit because the injectable loader and severity helper did not exist, and it passes on the fix commit. It covers recovery through refresh, a revoked sign-in returned as broken with no operator report, a busy lease, a still-rejected fresh token reported as a tenant fault, 503 staying alertable, a single tenant-level report for a terminal refresh, and the severity mapping.

Ran on cmux-lawrence-2: bun test on the new file plus the refresh, observability, responses-proxy, models-proxy, route-crash and Sentry suites (78 pass), bun run typecheck, bun run lint:complexity, and eslint on the changed files. ESLint reports one existing prefer-const finding in codexProxy.ts:834 that this PR does not touch. The full tests/coderouter-* glob has the same 6 cross-file mock failures on main and on this branch, and those files pass on their own.

Not verified against a live revoked credential.

Changelog

Fixed: coderouter now marks a revoked Codex sign-in as "Sign-in broken" after it fails a usage check, and stops polling it

Proof

Server-only change. The behavior tests above cover it.

View with [code]smith Autofix with [code]smith
Need help on this PR? Tag @codesmith-bot with what you need. Autofix is disabled.

Summary by cubic

Fixes the coderouter usage read so revoked Codex credentials are refreshed once instead of re-read forever, and surfaces revoked sign-ins as tenant faults.

  • A 401 on GET /api/coderouter/accounts forces one refresh and retries the read; a rotated token returns usage normally.
  • If another request already rotated past the expected revision, the forced refresh reuses that token instead of spending another refresh rotation.
  • A revoked sign-in is returned as state: "broken" so the dashboard shows "Sign-in broken" and later polls skip it.
  • Terminal refresh failures with revoked-sign-in codes (invalid_grant, expired, reused, revoked, credential_owner_mismatch) are reported once as fault: "tenant" warnings; other operator codes like invalid_client stay errors.
  • The proxy skips its duplicate provider_refresh report when the refresher already reported the broken credential.
  • A busy refresh lease defers the account to the next poll; provider outages on the usage endpoint stay alertable.

[!NOTE]
Medium Risk
Changes credential refresh, account state, and alerting paths for Codex usage and provider_refresh; misclassification could hide real outages or leave bad credentials polled, but behavior is covered by new tests and mostly aligns existing model-route logic.

Overview
Coderouter account usage polling now matches model-route recovery when Codex rejects stored tokens, and revoked team sign-ins are classified as tenant-owned faults so they stop paging operators.

On GET /api/coderouter/accounts, a 401 from the usage endpoint triggers one forced credential refresh and a retry. A rotated token returns usage; a terminal refresh marks the account broken for the dashboard; a busy lease surfaces credential_refreshing until the next poll. Forced refresh reuses a credential another request already rotated (higher revision) instead of burning another refresh rotation.

Observability: terminal refresh failures with revoked-sign-in codes report fault: "tenant" at warning in Sentry/PostHog via coderouterFailureSeverity. The Codex responses proxy skips a second provider_refresh report when CodeRouterCredentialBroken was already reported by the refresher. Usage still treats 503 and non-tenant failures as alertable.

Usage loading is refactored behind createAccountsUsageLoader (injectable deps); new tests cover refresh-on-401, broken/revoked paths, and severity mapping.

Reviewed by Cursor Bugbot for commit 6d7747116728d17bd3a404eaf33213fe98385bac. Bugbot is set up for automated code reviews on this repo. Configure here.

Summary by CodeRabbit

  • Bug Fixes
    • Account usage now retries once with refreshed credentials after an authentication failure, helping keep usage information current.
    • Accounts with broken or currently refreshing credentials now show distinct statuses instead of a generic unavailable result.
    • Authentication failures that persist after a refresh are classified separately from service or configuration errors, helping ensure alerts reflect the type of issue.