feat(evals): no per-provider concurrency cap by default

MI
4d ago
+282-51

Stack

Stacked on #3425 (which is on #3164). Review against that parent, not main.

Summary

#3087 gave every model provider a default width of 3 under -c. In practice -c 50 over three providers ran 9 wide, which is not what anyone asking for 50 expects. This removes the default: -c is the only limit unless you cap a provider.

  • Even split without a cap. Rows are interleaved in proportion to each provider's share of -c, so the slots split evenly (-c 50 over three providers: 17/17/16). Nothing holds a provider to its share: when one finishes early, the others take its slots, so the run stays -c wide.
  • Caps are opt-in. EVAL_PROVIDER_CONCURRENCY=anthropic=4 or evals config providers set anthropic concurrency 4 still cap a provider; the uncapped ones share what is left.
  • 429 backoff. A capped provider halves its cap, as before. An uncapped one has no cap to halve, so it halves what it was running when the 429 arrived (half of -c would usually be no backpressure at all). Both restore after 60 s.
  • Header and queue line show the split and which providers are capped:
  Env: BROWSERBASE  Harness: stagehand  Concurrency: 50 · anthropic ~17 · google ~17 · openai ~16
  running 50 · queued 0 · google 21 · openai 18 · anthropic 11

  Env: BROWSERBASE  Harness: stagehand  Concurrency: 20 · anthropic 2 (capped) · google ~9 · openai ~9
  running 20 · queued 59 · anthropic 2/2 · google 9 · openai 9

[!WARNING]
Behaviour change: runs that relied on the implicit 3-per-provider cap now run -c wide. CI's regression job (-c 25, three providers) goes back to 25 wide, as it was before #3087.

The doctor's heap check for in-process harnesses follows: an uncapped provider counts as -c sessions.

Test plan

  • vitest packages/evals/tests: 1259 passing (the three Playwright-backed integration files need a local browser)
  • tsc, oxlint (0 errors), fmt:check
  • New tests fail with their change reverted (8 mutation checks: default cap, remainder sharing, runner interleave weights, uncapped throttle, queue-line display)
  • End to end in a PTY against Browserbase, 120-run act matrix: -c 50 uncapped held running 50 and finished in 1m10s (2m35s at the old 9-wide); -c 20 with anthropic=2 held anthropic 2/2 with the other two sharing 18.

Summary by cubic

Removes the default 3-wide per-provider concurrency cap so -c is now the only limit unless a provider is explicitly capped. Rows interleave across providers in proportion to each provider's share of the cap, and uncapped shares add up exactly to -c (-c 50 over three providers splits 17/17/16, with the remainder going to the first ones); a provider that finishes early frees its slots for the others. Explicit caps via EVAL_PROVIDER_CONCURRENCY or providers.<p>.concurrency still apply; uncapped providers share what is left, and caps that add up to more than -c are scaled down so the first slots split in proportion. Throttling an uncapped provider halves what it was actually running (instead of the global cap) for 60 s. The run header and queue line now show the split, marking capped providers and sharing uncapped shares as approximations.

Behavior change: runs that relied on the implicit 3-per-provider cap now run -c wide.

Test plan

  • Unit tests cover default uncapped width, exact remainder sharing, interleave weight scaling, uncapped throttling, and queue-line display.
  • End-to-end run with -c 50 held 50 concurrent rows across three providers; -c 20 with anthropic=2 kept anthropic at 2 with the others sharing 18.