fix(agent): stop the run once when the provider does not serve the model
Merged
TH
9d ago+381+381+012345678901234567890123456789-26-26-01234567890123456789
Fixes ENG-985.
A GitHub Copilot Free login lists claude-haiku-4.5, gpt-5-mini, gpt-6-luna and the rest as enabled, and Copilot answers every call by id with 400 model_not_supported: Free serves models through auto model selection only (GitHub's plans page), and GET /models carries nothing that says so. The same body answers a model the settings turned off and an id Copilot does not list. The listing and the request path were both right. What was wrong is the failure mode: every test booted, called, and failed with MODEL_PROVIDER_FAILED and Copilot's bare sentence, the transport-error class, exit 3, and nothing named the cause.
copilot()(packages/e2e/src/oauth/copilot.ts) wraps its model: a 4xx whose body code ismodel_not_supportedis marked with the explanation a user reads, throughsrc/internal/unserved-model.ts. Copilot's other codes (unsupported_api_for_modelfor a model on the wrong endpoint) and every other provider answer stayMODEL_PROVIDER_FAILEDas before.- The agent reads only the mark:
unservedModelError(packages/e2e/src/agent/model/sdk.ts) turns a marked failure intoMODEL_UNAVAILABLE, from the judgment adapter and the act loop alike. No vendor code lives there; the next plan-gated provider marks its own answer. - The agent fixture reports that error through
WorkerModels.unservedModel, the hook the preflight already uses for a missing model: one run error, the run stops, exit 2, frome2e runande2e explorealike, and from a custom executor that callscopilot()itself. Two workers meeting it at once still produce one entry. - Docs: the
MODEL_UNAVAILABLErow and every page that defined the code as "no model configured" (Models, Debugging, Cache, the skill's agent and debugging references), the Subscriptions page (the listing shows GitHub's catalog, a plan refusal shows nowhere, Free serves models through auto selection only), the CLI reference, the skill's setup reference. No model names in any of it: which ids a plan serves is GitHub's to change.
flowchart LR
A[agent step calls copilot model] --> B{Copilot answer}
B -- 4xx model_not_supported --> C[copilot marks the failure with the explanation]
C --> D[agent: MODEL_UNAVAILABLE, fixture reports it once]
D --> E[run error, run stops, exit 2]
B -- any other failure --> F[MODEL_PROVIDER_FAILED, as before]
Verified
Ran it locally: yes
- testbed, built CLI, a Copilot Free login (
access_type_sku: free_limited_copilot), two agent tests (agent.assert, thenagent.act) withcopilot('claude-haiku-4.5'), then an id Copilot does not list, thene2e explore, thencopilot('gpt-4.1')as the control.
| main | this branch | |
|---|---|---|
e2e run, copilot('claude-haiku-4.5') | each test boots, calls, and fails: MODEL_PROVIDER_FAILED: model provider failed: The requested model is not supported.; exit 3 | one run error, the run stops in 2.0 s; 1 interrupted, 1 skipped; exit 2 |
e2e run, copilot('gpt-4.1-typo') | the same per-test failure | the same run error naming gpt-4.1-typo, 1.4 s, exit 2 |
e2e explore, copilot('claude-haiku-4.5') | the same per-step failure | no step ran, the same run error, 1.7 s, exit 2 |
e2e run, copilot('gpt-4.1') | passes | passes, 2 tests, 3 model calls |
this branch, copilot('claude-haiku-4.5')
✓ target "web" command ready 87ms
× copilot plan probe: assert on the todos page > agent.assert "one todo is listed and it is not marked done" 1.04s · 1 model call failed
❯ |web| tests-scratch/copilot-probe.e2e.ts (2 tests | 1 interrupted | 1 skipped) 1.61s ai 0 tokens
× copilot plan probe: assert on the todos page (interrupted) 1.61s ai 0 tokens
↓ copilot plan probe: act on the todos page [run interrupted before execution]
⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯ Run Errors 1 ⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯
ERROR configuration error MODEL_UNAVAILABLE
GitHub Copilot does not serve claude-haiku-4.5 to this account: the plan does not include it, it is turned off in Copilot settings on GitHub, or Copilot lists no such id; npx e2e models github-copilot lists the ids, and the listing cannot see plan limits
Test Files 1 interrupted | 0 passed (1)
Tests 1 interrupted | 0 passed | 1 skipped (2)
AI 0 tokens · 1 model calls · github-copilot.chat/claude-haiku-4.5
Errors 1 error
Duration 1.98s (startup 87ms)
this branch, e2e explore
✓ target "web" command ready 97ms
❯ |web| Exploring Open the todos page and look around
no step ran
ended: the run was cut short
⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯ Run Errors 1 ⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯
ERROR configuration error MODEL_UNAVAILABLE
GitHub Copilot does not serve claude-haiku-4.5 to this account: the plan does not include it, it is turned off in Copilot settings on GitHub, or Copilot lists no such id; npx e2e models github-copilot lists the ids, and the listing cannot see plan limits
Findings none
Steps 0 done of 1 · planning
AI 0 tokens · 1 model calls · github-copilot.chat/claude-haiku-4.5
Errors 1 error
Duration 1.70s (startup 97ms)
main, copilot('claude-haiku-4.5')
FAIL |web| tests-scratch/copilot-probe.e2e.ts > copilot plan probe: assert on the todos page
MODEL_PROVIDER_FAILED: model provider failed: The requested model is not supported.
❯ at http://127.0.0.1:4279/todos
❯ trace .e2e/results/copilot-probe-copilot-plan-probe-assert-on-the-79f031053e41dbdf/trace.md
Test Files 1 failed | 0 passed (1)
Tests 1 failed | 0 passed (1)
AI 0 tokens · 1 model calls · github-copilot.chat/claude-haiku-4.5
- Copilot's codes, same login,
curlagainst the API: a Responses-only model over/chat/completions, a typo'd id over either endpoint, and the Free plan's refusal all answermodel_not_supported; a chat model over/responsesanswersunsupported_api_for_model, which staysMODEL_PROVIDER_FAILED.