coderouter: single usage-read timeouts stay out of Sentry

Merged
LA
15d ago
+101-32

Summary

After https://github.com/manaflow-ai/cmux/pull/18392, CODEROUTER-WEB-18 and -1P still received about 63 Sentry events per hour, all at warning level with consecutive=1. A warning is still a Sentry event, so the change did not stop them.

The timeout rate also rose. Axiom, hourly usage reads (wham/usage) on 2026-10-07 UTC, as reads / p99 / reads at the 5 s cap:

Hour (UTC)Readsp99At 5 s cap
05:00 to 11:001,773 to 5,148 per hour1.0 to 2.1 s0 to 1 per hour
13:005,5043.3 s12
14:005,3353.2 s11
15:004,7693.7 s31
16:00 (partial)2,1885.0 s65

p50 stays at 0.31 to 0.41 s. OpenAI's usage endpoint tail has degraded since about 13:00 UTC, and the 5 s budget now cuts about 1 to 3% of reads, all stalled bodies after 200 headers. I kept the 5 s budget. The account list waits for its slowest read, and the next poll retries anyway.

A usage-read timeout with consecutive below 3 no longer calls the reporter. It writes one coderouter.usage_timeout log line and a usage_timeout span on the route trace with provider, timeout and consecutive. Only an account that times out on 3 polls in a row reports to Sentry, as an error.

Testing

The red commit rewrote the timeout tests to require no report for a single timeout. It failed 3 tests on main code and passes with the fix. On cmux-lawrence-2, the usage-timeouts, rejected-credential, module-imports, accounts-route and request-telemetry suites ran 122 pass, 0 fail. bun run typecheck and bun run lint:complexity pass, and ESLint finds no errors.

Changelog

none

Proof

Server-only telemetry change. The behavior tests above cover it.


View with [code]smith Autofix with [code]smith
Need help on this PR? Tag @codesmith-bot with what you need. Autofix is disabled.


Summary by cubic

Single usage-read timeouts in coderouter no longer reach Sentry: only accounts that time out on 3 consecutive polls report, as an error. Earlier timeouts write a coderouter.usage_timeout log line and a usage_timeout span on the route trace with provider, timeout, and consecutive.

  • Credential refresh timeouts keep their normal Sentry report and never count toward the streak.
  • A successful read resets the streak; the 5 s read budget is unchanged.

[!NOTE]
Low Risk
Server-only observability change; account responses still surface usageError: "timeout" while reducing Sentry noise until three consecutive failures.

Overview
Coderouter usage polling no longer sends every stalled usage read to Sentry. Isolated timeouts (below the existing streak of 3) only run a new observeTimeout path: a coderouter.usage_timeout log and a usage_timeout route span via recordCoderouterSpan, with no reportCoderouterFailure call.

Sentry reporting is limited to accounts that hit USAGE_TIMEOUT_ERROR_STREAK consecutive usage-read timeouts, still as a provider_usage error (no fault: "upstream"). Fetch, retry-after-401, and response.json() are wrapped in usageRead, which rethrows timeouts as UsageReadTimeout so credential-refresh timeouts keep their normal report and do not advance the streak.

Tests now assert observe-vs-report behavior and the refresh-timeout exception.

Reviewed by Cursor Bugbot for commit ef3a0af390d71d6f83e0961de3ebfa9e2219eda9. Bugbot is set up for automated code reviews on this repo. Configure here.

Summary by CodeRabbit

  • Bug Fixes
    • Usage read timeouts—including delays while reading a response—are handled consistently across initial requests and retries.
    • A usage timeout is reported after three consecutive timeouts; a successful read resets the count. Isolated timeouts do not trigger a report.
    • Credential-refresh timeouts remain separate and do not affect the usage-timeout count.
    • Timeout events are logged and included in route tracing, without being classified as provider faults.