PR #18392 - manaflow-ai/cmux

Summary

Two misclassified coderouter Sentry errors after https://github.com/manaflow-ai/cmux/pull/18297.

CODEROUTER-WEB-18 and CODEROUTER-WEB-1P now fire only with TimeoutError from the account-list usage read (chatgpt.com/backend-api/wham/usage, 5 s budget for headers and body). Axiom, 2026-10-06 14:00 to 2026-10-07 15:00: 52,316 reads, p50 0.34 s, p95 0.91 s, p99 2.2 s, and 20 reads at the 5 s cap. In those 20, the provider sent 200 headers and then stalled the body. Traces 5cee9a27… and ea3ac973… show this on two teams, with one account's read hanging while the others finished in about 0.3 s. The budget covers normal latency. The fault is a rare provider-side stall that the next poll retries. Raising the budget would only hold the whole account list longer, because the list waits for its slowest read.

A usage-read timeout now reports as an upstream fault at Sentry and PostHog warning, with timeout and consecutive context, and the account returns usageError: "timeout". When the same account times out on 3 consecutive polls, it reports as an error. The streak counter is per instance and bounded, and a successful read resets it.

CODEROUTER-WEB-1Q is a terminal provider_refresh from the usage-read 401 path (trace a163aec5…: 3 usage 401s, then a 401 from /oauth/token) that still reached Sentry as an error. The likely cause is that the revoked-sign-in classifier missed OpenAI's refresh_token_invalidated, which the Codex CLI maps to "revoked" (logout or password change). It now joins refresh_token_expired and refresh_token_reused as a tenant fault. invalid_client and other configuration codes stay operator errors. The refresh report now carries a sanitized failure_code, so the next event shows the exact code.

Testing

tests/coderouter-usage-timeouts.test.ts failed on the red commit (the streak tracker did not exist) and passes on the fix. It covers a single timeout as a warning, escalation at 3 consecutive timeouts, reset on success, upstream severity, refresh_token_invalidated as tenant with its code reported, and invalid_client staying an error. On cmux-lawrence-2, those tests plus the rejected-credential, refresh, observability, responses-proxy and api-key-provider suites ran 81 pass, 0 fail. bun run typecheck and bun run lint:complexity pass, and ESLint finds no errors on the changed files.

Not proven: the exact code behind CODEROUTER-WEB-1Q. Telemetry did not record it before this change, and I did not read prod DB rows.

Changelog

none

Proof

Server-only telemetry change. The behavior tests above cover it.

^{Need help on this PR? Tag @codesmith-bot with what you need. Autofix is disabled.}

Summary by cubic

Fixes misclassified coderouter Sentry errors so transient usage-read timeouts warn instead of paging, and revoked sign-ins are tenant faults.

A usage-read timeout now reports as an upstream warning, and only escalates to an error after 3 consecutive polls for the same account. The streak counter is per loader (one tracker per loader by default), bounded, and a successful read resets it. refresh_token_invalidated now joins refresh_token_expired and refresh_token_reused as a tenant fault, while configuration codes like invalid_client stay operator errors; the refresh report also carries a sanitized failure_code with the exact code.

[!NOTE]
Low Risk
Server-only observability and fault tagging; runtime usage/refresh behavior is unchanged aside from timeout reporting and refresh failure context.

Overview
Reclassifies coderouter failure telemetry so rare provider usage-read stalls stop paging operators, while revoked OAuth sign-ins are treated as tenant-owned issues.

Usage reads that hit the 5s TimeoutError now return usageError: "timeout" and report provider_usage with { fault: "upstream" } (Sentry/PostHog warning). A per-instance consecutive-timeout streak escalates to a full error only after 3 polls for the same account; a successful read clears the streak. coderouterFailureSeverity now downgrades any explicit fault (tenant or upstream) to warnings.

Credential refresh failures add a sanitized failure_code tag, and OpenAI’s refresh_token_invalidated is recognized as a revoked sign-in (tenant fault) alongside the existing expired/reused codes; codes like invalid_client remain operator errors.

^{Reviewed by Cursor Bugbot for commit 97a468f378d9cd4a41b88ce2241dac151c5df7bc. Bugbot is set up for automated code reviews on this repo. Configure here.}

Summary by CodeRabbit

Bug Fixes
- Usage checks now distinguish timeouts from other availability failures. Consecutive timeouts are tracked per account, and a successful check resets the count.
- Provider and credential-refresh failures are classified more consistently, including invalidated sign-in credentials. Unrecognized failure codes are handled safely.
- Repeated usage timeouts receive different severity reporting from initial timeouts, helping distinguish transient issues from ongoing ones.