fix(bedrock): non-empty Converse streams in the release build and cached tokens billed once

AN
1mo ago
+636-10

Issue for this PR

Closes #1423

Type of change

  • Bug fix
  • New feature
  • Refactor / code improvement
  • Documentation

What does this PR do?

Two Amazon Bedrock correctness fixes, in separate commits. A third Bedrock problem, requests that stall with no response and no error, is being handled separately and is not touched here.

Commit 1: empty Converse streams in the built binary

Why: in a binary built by packages/opencode/script/build.ts, every Bedrock Converse stream ended with finish reason other, zero tokens, no text and no error. Running from source was fine.

Root cause (confirmed): the build bundles with conditions: ["browser"] and a Bun target. Under that pair @smithy/core/serde resolves to index.browser.js, where fromString and fromArrayBuffer are Symbol.for("node-only") placeholders. @smithy/util-buffer-from (4.x) re-exports them, @smithy/util-utf8 calls them, and @ai-sdk/amazon-bedrock's event-stream decoder wraps the decode in try { ... } catch { break }, so the TypeError disappears. Measured: bundling the same entry with the browser condition works with a browser target and returns an empty stream with a bun target; I attribute that to the browser field of @smithy/util-utf8 being applied only for the former, which I did not separately prove.

Change: new script/smithy-node-serde-plugin.ts, registered in build.ts. It redirects only @smithy/core/serde when imported from inside @smithy/util-buffer-from to the package's own Node build (the file a source run uses). No other import changes resolution. The earlier benchmark workaround replaced @smithy/util-buffer-from outright with a hand-written Buffer shim for every importer; this uses the real implementation and a narrower match.

Commits 2 and 3: cached tokens billed twice

Why: reported cost on the Bedrock Converse path was about four times too high.

Root cause (confirmed on current main, not only reported): AI SDK v6 adapters for Anthropic and Bedrock report usage.inputTokens as noCache + cacheRead + cacheWrite (Bedrock's raw inputTokens excludes cache; the adapter adds it back) and put the uncached part in usage.inputTokenDetails.noCacheTokens. Session.getUsage still treated inputTokens as excluding cache whenever anthropic or bedrock provider metadata was present, so each cached token was charged at the full input price in addition to its cache read or write price. A recorded Bedrock step: inputTokens 61363, cache write 61359, so about 4 uncached tokens, but 61363 were charged as input. The same applies to direct Anthropic (same adapter convention) and to the learn/import/bootstrap accounting (accountUsage), which forwarded the inclusive number.

Change: on the Anthropic/Bedrock branch getUsage uses inputTokenDetails.noCacheTokens when the SDK supplies it (and cacheWriteTokens as a last-resort write count); accountUsage forwards both. The cache-read count falls back to inputTokenDetails.cacheReadTokens when cachedInputTokens is absent, the experimentalOver200K tier check now counts the whole prompt (uncached + cache read + cache write), and the generation telemetry tokens_cache_read field follows the same fallback. Everything else is unchanged: OpenAI-style providers still subtract the cached part, and raw counts without details keep the old arithmetic.

Behaviour a user could notice: for Anthropic-family models with prompt caching, reported tokens.input, tokens.total, session cost and the telemetry tokens_input drop to the true values. The 200K-context price tier is selected from the whole prompt (uncached + cache read + cache write).

Falsifiable before and after (live, haiku, amazon-bedrock/us.anthropic.claude-haiku-4-5-20251001-v1:0, source run, same prompt): before, input 48064, cache.read 48062, cost 0.05818; with the fix the same shape costs (2 x 1.1 + 4 x 5.5 + read x 0.11)/1e6 exactly, and the built binary reported input 2, cache.write 37819, cost 0.052025325, which equals (2 x 1.1 + 4 x 5.5 + 37819 x 1.375)/1e6 using the catalogue prices.

Deployment readiness: self-contained; no migration, configuration or credential change. The build fix takes effect in the next release build; the cost fix on upgrade.

Tenant and user impact: users of built binaries with Bedrock Converse get responses instead of empty ones (with environment-key or bearer-token auth; see limits). Users of Anthropic-family models with caching see lower, correct reported cost and token input counts. No effect on other providers.

Known limits and things not changed:

  • The AWS credential chain has the same placeholder problem for profile-file and web-identity credentials in the release binary (parseKnownFiles from @smithy/core/config is a placeholder under the same conditions). Environment keys and bearer tokens work and are what was tested live. This PR does not fix it; doing so means redirecting more @smithy/core subpaths for many importers. A reviewer reproduced it with a bundle probe, and I reproduced it too (source run resolves a shared credentials file; the plugin-built bundle throws).
  • src/session/session.ts (the Effect-based getUsage) already subtracts unconditionally and is unchanged.

How did you verify your code works?

Unit (no network, no credentials):

  • test/script/bedrock-eventstream-build.test.ts bundles a probe that decodes a synthetic application/vnd.amazon.eventstream body through the real @ai-sdk/amazon-bedrock provider, using the release build's conditions: ["browser"] and a Bun target, then runs the bundle. With the plugin: text hello, finish stop, tokens 4/2. Without it (control): empty text, finish other. A further test pins that build.ts registers the plugin (this one is a source-text check) and two tests pin the plugin's scope (only @smithy/util-buffer-from importers are redirected). It would have caught the defect because the decode path is exercised under the exact resolution conditions where the placeholder is used; a source-only test cannot, since source runs do not use the browser condition. Without the build wiring the wiring test fails.
  • test/session/session-getusage-inclusive.test.ts: usage shapes taken from recorded Bedrock step events (counts only): no caching, cache write, cache read, mixed, direct Anthropic, OpenAI-style unchanged, the non-Anthropic guard, raw counts unchanged, and accountUsage with a generateObject shape; each asserts exact cost. Against the previous session/index.ts and learn/usage.ts 5 of 9 fail (the cases with caching); the no-caching and unchanged-provider cases pass before and after by design.
  • Also run: existing session-getusage, test/server/negative-tokens-regression, test/altimate/learn (2400 passed, 0 failed across test/altimate/learn, processor, telemetry and getUsage suites on the final head; CI TypeScript job also green on the previous head), bun run typecheck clean, script/upstream/analyze.ts --markers --base main --strict clean.

Integration with real components (built binary, real Bedrock, us-east-1, tiny prompts):

  • Binary built from current main for darwin-arm64: step finish other, 0 tokens, no text.
  • Binary built with this change, same platform: ok, finish stop, non-zero tokens, for Haiku 4.5 and for us.anthropic.claude-sonnet-5-5.
  • Source run (no build) is not affected by the empty-stream defect: it returned text.

Not run: Linux and Windows binaries. build.ts applies one Bun.build configuration to every target, and module resolution does not depend on the OS, so the same defect and fix are expected there, but I only built and ran darwin-arm64. The npm-installed binary on macOS is the same build output. Remaining risk: a Linux-only difference in the compiled binary. Direct Anthropic, Vertex-Anthropic and Mantle (OpenAI Responses) were not called live; Anthropic and Vertex-Anthropic are covered by unit tests only.

Screenshots / recordings

Not applicable (no UI change).

Checklist

  • I have tested my changes locally
  • I have not included unrelated changes in this PR

🤖 Generated with Claude Code

[!NOTE]
Medium Risk
Changes release bundling resolution and session cost/telemetry logic for Anthropic-family providers; well-covered by new tests but affects billing and Bedrock streaming in shipped binaries.

Overview
Fixes Amazon Bedrock in release binaries and Anthropic/Bedrock token billing after AI SDK v6 usage shape changes.

Release build: Adds smithyNodeSerdePlugin to the Bun.build pipeline (conditions: ["browser"]). It forces @smithy/core/serde to resolve to the Node implementation when imported from @smithy/util-buffer-from, so Bedrock Converse event-stream decoding no longer hits browser-only placeholders and returns empty streams. Regression tests bundle a synthetic event-stream probe with and without the plugin.

Usage & cost: Session.getUsage treats AI SDK v6 inclusive inputTokens on Anthropic/Bedrock by preferring inputTokenDetails.noCacheTokens, with guarded fallbacks for cache read/write in inputTokenDetails. Learn accountUsage forwards those detail fields; long-context tiering uses inputTotal (uncached + cache read + write). Generation telemetry emits tokens_cache_read only when cacheReadReported agrees with accepted accounting.

Types: GenerateUsage gains optional noCacheTokens in inputTokenDetails.

Reviewed by Cursor Bugbot for commit 94df9bc5c3aee5b4cf1ab056e61b68022130c620. Bugbot is set up for automated code reviews on this repo. Configure here.

Summary by CodeRabbit

  • Bug Fixes
    • Improved Amazon Bedrock response handling so streamed text, completion reasons, and token counts are reported correctly.
    • Improved token usage accounting across supported providers, including cache-read and cache-write tokens, when calculating input usage and pricing.
    • Corrected long-context pricing eligibility to account for total input tokens, including cache reads and writes.
    • Improved cache-read token reporting in usage telemetry, including when providers report cache usage through detailed token counts.