PR #3110 - mnfst/llm-gateway

Fixes #3107. Also addresses item 4 of #3103 (Astra rejecting none).

Summary

Bedrock Runtime Chat Completions rejects function tools next to any reasoning effort but none for the catalogued GPT profiles (openai.gpt-6-luna, -sol, -astra, openai.gpt-5.6-luna). Astra rejects none as well, so it cannot take function tools on chat at all. Runtime Responses accepts the same combination.

This PR sends exactly those chat requests to Runtime Responses and leaves everything else where it is. It is a narrow, catalog-driven route, it can be switched off with an environment variable, and it was checked against real Bedrock.

The problem

#3075 forwards the caller's reasoning_effort and sends none when function tools come without one. That fixes tools-without-effort for Luna and Sol. Two cases are still broken on main:

A caller that sends an effort together with tools still gets a 400 (for example OpenCode, which sends max): Function tools with reasoning_effort are not supported for global.openai.gpt-6-luna in /v1/chat/completions. To use function tools, use /v1/responses or set reasoning_effort to 'none'.
Astra with function tools fails whatever the effort: the injected none is rejected (Unsupported value: 'none' is not supported with this model), and any other effort is rejected next to tools.

Moving those requests to Runtime Responses exposed two problems in the Responses to Chat Completions stream conversion, which this PR also fixes (see below).

What changed, and why

Two commits, each independent.

Commit 1: chatgpt-adapter.ts (stream conversion)

Change Why

Read the event name from data.type when a frame has no event: line Bedrock Runtime sends Responses frames with no event: line. Without this the converted stream came out empty.

Number streamed tool calls densely from 0, in first-seen order per stream (denseIndex) A reasoning item takes output_index 0, so the first function call was streamed with tool_calls[].index 1 and no index 0. The official SDKs fail on that: openai for Node 7.47.0 throws Cannot read properties of undefined (reading 'id') from .stream(), and the Python SDK raises IndexError. Without stream state the raw output_index is kept, so existing callers and specs behave as before.

Change	Why
Read the event name from `data.type` when a frame has no `event:` line	Bedrock Runtime sends Responses frames with no `event:` line. Without this the converted stream came out empty.
Number streamed tool calls densely from 0, in first-seen order per stream (`denseIndex`)	A reasoning item takes `output_index` 0, so the first function call was streamed with `tool_calls[].index` 1 and no index 0. The official SDKs fail on that: `openai` for Node 7.47.0 throws `Cannot read properties of undefined (reading 'id')` from `.stream()`, and the Python SDK raises `IndexError`. Without stream state the raw `output_index` is kept, so existing callers and specs behave as before.

Commit 2: routing

Change	Why
`routing/function-tools.ts`: one `hasFunctionTools(tools)`; `sanitizeOpenAiBody` now calls it	The #3075 injection and the new decision need the same definition of "function tools". Behaviour-preserving refactor.
`rejectsReasoningNone?: true`, set on Astra only, and a pure `bedrockRuntimeChatNeedsResponses(model, body)` in `bedrock-runtime-capabilities.ts`	The decision is a function of the catalog entry and the request: catalogued GPT profile, function tools present, and either the model rejects `none` or the effort is a string other than `none`. "Is GPT" is `chatTokenParameter === 'max_completion_tokens'`, the same test #3075 uses, on purpose.
`acceptsReasoningSummary` on the `bedrock-runtime-responses` endpoint, used by `mapReasoningEffort`	Maps `reasoning_effort` to `reasoning: { effort, summary: 'auto' }`. It is an endpoint flag, not a check on the endpoint key, because a stored-region connection reaches `buildRequest` as endpoint `custom`. The regional override copies the template, so it gets the flag.
The swap happens inside `ProviderClient.resolveEndpoint` (default connection through `resolveBedrockEndpointKey(model, 'responses')`, stored-region connection through `buildEndpointOverride`)	`isChatGpt`, `wireFormat` and `wireApiMode` derive from the endpoint, so the response handler, the Autofix retry (`retryWireBody`) and the recorder follow it without changes. `resolveBedrockEndpointKey` and the forward-endpoint resolver are untouched.
`BEDROCK_RUNTIME_CHAT_VIA_RESPONSES` kill switch	Only the exact value `false` turns the route off; it is read on every call, so no rebuild is needed.
`CLAUDE.md` and `.env.example` lines, changeset	The variable is documented where the others are.

Production code: 69 lines of code and 27 of comments across 6 files. The rest of the diff is tests.

How it was done

Test first. The specs were written before the implementation and checked to fail for the right reason (behaviour, not type errors): 274 red tests, 36 guard tests that must stay green.
Four new spec files cover the adapter (frames without event:, dense indexing), the routing predicate, ProviderClient (default and stored-region connection, kill switch, second turn, Autofix retry) and the fallback chain with a tier-supplied effort. provider-client.spec.ts drops one variant that asserted the old behaviour: luna with tools and high staying on chat. The new spec covers that case.
The conversion itself is not new: it is toResponsesRequest, the same chat to Responses conversion the Mantle route already applies to Bedrock GPT. Differences: the Runtime route also maps reasoning_effort and sends Manifest's hashed prompt_cache_key; Mantle does neither and keeps dropping the effort.

What does not change

Requests without function tools, requests with effort none or no effort on Luna/Sol (the #3075 path), Kimi, base model ids (Mantle) and non-Bedrock providers keep their route.
/v1/responses and /v1/messages requests keep their paths.
No change to Autofix, the fallback chain, recording or the dashboard.

How this differs from the earlier attempts

#3066 (closed: "we'd rather avoid rerouting every Bedrock GPT call through the Responses translation, too much surface for this one bug"). That PR sent every chat and Messages request for the catalogued GPT profiles to Responses, tools or not. This one does not:

	#3066	This PR
Which requests move	All chat and Messages requests for the 4 catalogued GPT profiles	Only chat requests with function tools plus an effort the chat route rejects (or Astra with tools)
Requests that work today	Moved too	Not touched. The route only takes requests that return 400 on `main` (checked on real Bedrock)
`/v1/messages`	Rerouted	Not rerouted
Catalog	A flag on every GPT entry	One optional field on one entry, plus a pure predicate that reuses #3075's GPT test
Rollback	None	`BEDROCK_RUNTIME_CHAT_VIA_RESPONSES=false`
Relation to #3075	An alternative to it	Builds on it. Its behaviour is pinned by guard tests

#3067 (closed: the 400 is the model's real limit, handle it in Autofix rather than keep a model list in the gateway). It is a different problem (assistant prefill), and I am not retrying it. The same objection can be raised here, so I want to be direct: this PR does add a per-model fact to the catalog (rejectsReasoningNone, one entry). The catalog already carries such facts (chatTokenParameter, apis), and the new field is read by one pure function. I have not tried to do this in Autofix: an Autofix repair can only change the request, and the only change that makes the chat route accept these requests is dropping or lowering the reasoning the caller asked for. I have not checked what Phoenix returns for this error.

#3075. Its description says an explicit effort with tools "is the caller's choice and fails the same way it does on OpenAI, where Autofix handles it", and that it was not run against live Bedrock. I did run these cases on real Bedrock, which is why this PR exists: the case still fails there, and Astra is not fixed by #3075. If you prefer to keep that stance, this PR is easy to drop; the adapter commit stands on its own.

Why this could be accepted now

The route is narrow and explainable: function tools together with an effort the chat route rejects, on catalogued GPT profiles. Everything else is covered by guard tests that assert it did not move.
It only moves requests that already fail with 400, so no request that works today changes route.
It reuses the existing conversion and the existing buildEndpointOverride, adds no new endpoint type and no new provider, and the production diff is small.
There is a kill switch.
The two adapter fixes are bugs on their own. Frames without event: produced an empty stream, and index 1 without 0 breaks the official SDKs. They are in a separate commit that does not depend on the routing.
It was exercised the way a client uses it, not only through unit tests (results below).

Risks and limits

Model and cost change for some users. A request that fails with 400 today and is rescued by a fallback model or by Autofix will now succeed on the GPT profile itself, with reasoning. The served model, the output tokens and the cost change for those users.
Parameters the conversion drops on this route: temperature, top_p, stop, seed, n, presence_penalty, frequency_penalty, logit_bias, user, metadata, stream_options, and a client-supplied prompt_cache_key. These requests fail today, so nothing that worked is lost, but a client that depends on stop or seed will not get them.
With no system message, instructions defaults to "You are a helpful assistant.".
qualifyEmptyChatCompletion does not apply to the Responses route, so an empty completion comes back as 200 with content: null.
Reasoning from earlier turns is not replayed.
Shared adapter. The index change also applies to the other Responses-backed routes (OpenAI, Copilot, xAI, ChatGPT subscription, Mantle): their streamed tool calls now start at index 0 when a reasoning item comes first. Before, they started at 1 with no 0. The event: paths are covered by regression tests, and SDKs failed on the old output, so I expect this to be an improvement, but it is a visible change.
Astra with an explicit none still returns 400, because Astra rejects none on Responses too.
The endpoint is decided in two places: the forward-endpoint resolver picks the Runtime chat endpoint without seeing the body, and resolveEndpoint swaps it when the body qualifies. Both are commented and tested for the default and the stored-region connection.
Known gap, older than this PR: the Responses to chat stream transformer never sends delta.role: "assistant" in the first chunk. Clients built on the Vercel AI SDK (such as OpenCode) are not affected; the OpenAI Node SDK .stream() helper is. That is a separate change.
Acceptance risk: this is a second attempt at an approach that was closed once, so the maintainers may well prefer to keep the status quo and leave it to Autofix.

Testing

Real Bedrock (us-east-1, 2026-11-03). Same request against main and this branch:

Case	`main`	This PR
luna, tools + `high`	400	200, tool call with reasoning (430 chars non-stream, 1088 stream), tool call index 0
luna, tools + `max` and `xhigh`, `global.` and `us.` profiles	400	200, tool call with reasoning
astra, tools, no effort (stream and non-stream)	400 (`none` unsupported)	200
astra, tools + `high`	400	200
luna and astra, second turn with a tool result	400	200, final text
luna, tools + `high` + `temperature` + `stop`	400 (`stop` unsupported)	200
luna, `tool_choice: required` + `parallel_tool_calls: false`	400	200, tool call
luna, `tool_choice: none`	400	200, no tool call
kimi-k3, tools + `high`	200	200, unchanged (chat)
astra, tools + `none` (exploratory)	400	400 (Astra rejects `none` on Responses too)

Client stacks.

The official openai SDKs with the real transformer: before, the first tool call at index 1 broke .stream() on Node 7.47.0 and on Python 3.42.1. After, both accumulate it. Frames without event: used to produce an empty stream.
An agent loop with the Vercel AI SDK (ai 5.0.285 + @ai-sdk/openai-compatible 1.0.61, the stack OpenCode uses; streamText, a tool, providerOptions.reasoningEffort): on main, global.openai.gpt-6-luna with max fails with the 400 above. With this PR the loop completes: tool call, tool result, final answer, reasoning streamed. Same for Luna high and Astra high and max.

Suites.

Backend unit: 521 suites, 9932 tests, all passing. Every changed statement is covered; one pre-existing branch in the adapter (the : 0 fallback when output_item.added has no output_index) is not.
shared 489 and cli 325 pass. tsc, eslint (0 errors) and prettier are clean, and the changeset check passes.
Failures that are not caused by this PR: one frontend test (Account.test.tsx expects the text "August 15, 2026", a date that has passed) and the e2e tests ST-07 and ST-08 (limits and notifications). ST-07 and ST-08 also fail on an unmodified main at 394d9a94a, 3 runs out of 3.

Summary by cubic

Fixes #3107 and item 4 of #3103. Bedrock Runtime Chat Completions rejects function tools next to a reasoning effort for the catalogued GPT profiles (openai.gpt-6-luna, -sol, -astra, openai.gpt-5.6-luna), and Astra rejects even none, so those requests now route through the Runtime Responses API, which accepts the same combination.

Route

Only requests that already 400 on Chat Completions change route; everything else keeps its path (Kimi, base model ids, effort none on Luna/Sol, no tools, Messages, native Responses).
Disable with BEDROCK_RUNTIME_CHAT_VIA_RESPONSES=false, read on every call; this also reverts the Astra function-tools route.
reasoning_effort maps to reasoning: { effort, summary: 'auto' }; sampling parameters (temperature, top_p, stop, seed, ...) are dropped and not forwarded.

Fixes and side effects

The Responses-to-Chat stream conversion now reads frames whose event name lives only in data.type (Bedrock Runtime sends no event: line), which previously produced an empty stream, and numbers tool calls densely from 0 in first-seen order — a reasoning item takes output index 0, so the first tool call at index 1 broke the official openai Node and Python SDKs. Frames with no output_index default to index 0.
Tool calls now number from 0 on every other Responses-backed route (OpenAI, Copilot, xAI, ChatGPT subscription, Mantle) when reasoning precedes them — visible, but the old output broke the official SDKs.
Requests that previously failed and were rescued by Autofix or a fallback model now succeed with reasoning on the GPT profile itself, changing served model, output tokens, and cost for those callers.
Astra with an explicit none effort still 400s, because Astra rejects none on Responses too.