fix(agent): stop the run once when the provider does not serve the model

Merged
TH
9d ago
+381-26

Fixes ENG-985.

A GitHub Copilot Free login lists claude-haiku-4.5, gpt-5-mini, gpt-6-luna and the rest as enabled, and Copilot answers every call by id with 400 model_not_supported: Free serves models through auto model selection only (GitHub's plans page), and GET /models carries nothing that says so. The same body answers a model the settings turned off and an id Copilot does not list. The listing and the request path were both right. What was wrong is the failure mode: every test booted, called, and failed with MODEL_PROVIDER_FAILED and Copilot's bare sentence, the transport-error class, exit 3, and nothing named the cause.

  • copilot() (packages/e2e/src/oauth/copilot.ts) wraps its model: a 4xx whose body code is model_not_supported is marked with the explanation a user reads, through src/internal/unserved-model.ts. Copilot's other codes (unsupported_api_for_model for a model on the wrong endpoint) and every other provider answer stay MODEL_PROVIDER_FAILED as before.
  • The agent reads only the mark: unservedModelError (packages/e2e/src/agent/model/sdk.ts) turns a marked failure into MODEL_UNAVAILABLE, from the judgment adapter and the act loop alike. No vendor code lives there; the next plan-gated provider marks its own answer.
  • The agent fixture reports that error through WorkerModels.unservedModel, the hook the preflight already uses for a missing model: one run error, the run stops, exit 2, from e2e run and e2e explore alike, and from a custom executor that calls copilot() itself. Two workers meeting it at once still produce one entry.
  • Docs: the MODEL_UNAVAILABLE row and every page that defined the code as "no model configured" (Models, Debugging, Cache, the skill's agent and debugging references), the Subscriptions page (the listing shows GitHub's catalog, a plan refusal shows nowhere, Free serves models through auto selection only), the CLI reference, the skill's setup reference. No model names in any of it: which ids a plan serves is GitHub's to change.
flowchart LR
  A[agent step calls copilot model] --> B{Copilot answer}
  B -- 4xx model_not_supported --> C[copilot marks the failure with the explanation]
  C --> D[agent: MODEL_UNAVAILABLE, fixture reports it once]
  D --> E[run error, run stops, exit 2]
  B -- any other failure --> F[MODEL_PROVIDER_FAILED, as before]

Verified

Ran it locally: yes

  • testbed, built CLI, a Copilot Free login (access_type_sku: free_limited_copilot), two agent tests (agent.assert, then agent.act) with copilot('claude-haiku-4.5'), then an id Copilot does not list, then e2e explore, then copilot('gpt-4.1') as the control.
mainthis branch
e2e run, copilot('claude-haiku-4.5')each test boots, calls, and fails: MODEL_PROVIDER_FAILED: model provider failed: The requested model is not supported.; exit 3one run error, the run stops in 2.0 s; 1 interrupted, 1 skipped; exit 2
e2e run, copilot('gpt-4.1-typo')the same per-test failurethe same run error naming gpt-4.1-typo, 1.4 s, exit 2
e2e explore, copilot('claude-haiku-4.5')the same per-step failureno step ran, the same run error, 1.7 s, exit 2
e2e run, copilot('gpt-4.1')passespasses, 2 tests, 3 model calls
this branch, copilot('claude-haiku-4.5')
 ✓ target "web" command ready 87ms
   × copilot plan probe: assert on the todos page > agent.assert "one todo is listed and it is not marked done" 1.04s · 1 model call failed
 ❯ |web| tests-scratch/copilot-probe.e2e.ts (2 tests | 1 interrupted | 1 skipped) 1.61s ai 0 tokens
   × copilot plan probe: assert on the todos page (interrupted) 1.61s ai 0 tokens
   ↓ copilot plan probe: act on the todos page [run interrupted before execution]

⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯ Run Errors 1 ⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯

 ERROR  configuration error MODEL_UNAVAILABLE
GitHub Copilot does not serve claude-haiku-4.5 to this account: the plan does not include it, it is turned off in Copilot settings on GitHub, or Copilot lists no such id; npx e2e models github-copilot lists the ids, and the listing cannot see plan limits

 Test Files  1 interrupted | 0 passed (1)
      Tests  1 interrupted | 0 passed | 1 skipped (2)
         AI  0 tokens · 1 model calls · github-copilot.chat/claude-haiku-4.5
     Errors  1 error
   Duration  1.98s (startup 87ms)
this branch, e2e explore
 ✓ target "web" command ready 97ms
 ❯ |web| Exploring  Open the todos page and look around
   no step ran
   ended: the run was cut short

⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯ Run Errors 1 ⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯

 ERROR  configuration error MODEL_UNAVAILABLE
GitHub Copilot does not serve claude-haiku-4.5 to this account: the plan does not include it, it is turned off in Copilot settings on GitHub, or Copilot lists no such id; npx e2e models github-copilot lists the ids, and the listing cannot see plan limits

   Findings  none
      Steps  0 done of 1 · planning
         AI  0 tokens · 1 model calls · github-copilot.chat/claude-haiku-4.5
     Errors  1 error
   Duration  1.70s (startup 97ms)
main, copilot('claude-haiku-4.5')
 FAIL  |web| tests-scratch/copilot-probe.e2e.ts > copilot plan probe: assert on the todos page
MODEL_PROVIDER_FAILED: model provider failed: The requested model is not supported.
 ❯ at http://127.0.0.1:4279/todos
 ❯ trace .e2e/results/copilot-probe-copilot-plan-probe-assert-on-the-79f031053e41dbdf/trace.md

 Test Files  1 failed | 0 passed (1)
      Tests  1 failed | 0 passed (1)
         AI  0 tokens · 1 model calls · github-copilot.chat/claude-haiku-4.5
  • Copilot's codes, same login, curl against the API: a Responses-only model over /chat/completions, a typo'd id over either endpoint, and the Free plan's refusal all answer model_not_supported; a chat model over /responses answers unsupported_api_for_model, which stays MODEL_PROVIDER_FAILED.