test: hermetic e2e suite for Altimate Base (registration, catalog, inference, rate-limit/budget, error surfacing)

Merged
AN
2mo ago
+2090-13

Issue for this PR

Closes #1247

Type of change

  • New feature
  • Bug fix
  • Refactor / code improvement
  • Documentation

What does this PR do?

Adds 53 hermetic e2e tests across 5 suites for Altimate Base's free-tier client (src/altimate/free/*), consolidated onto the shared _fixtures/altimate-base-harness.ts + _fixtures/fake-gateway.ts harness already on this branch:

  • altimate-base-registration-gaps.test.ts (11) — HTTP 4xx/5xx/network/malformed-JSON register failure mapping onto RegistrationError, the exact request payload (hashed install secret, cli_version), and retry idempotency.
  • altimate-base-catalog.test.ts (9) — model catalog / provider isolation (Provider.list() / Provider.defaultModel() / Provider.sort()) against a credential minted through the real consent + register path.
  • altimate-base-inference-e2e.test.ts (5) — full register → provider-list → authorizedFetch round trip, plus the placeholder-vs-real-key and credential-storage isolation properties.
  • altimate-base-rate-limit-messages.test.ts (21) — every describeRateLimit / describeRequestTooLarge branch: per-minute token throttle, burst throttle, wallet/global/unknown budget, request-too-large byte math, and the malformed/unrecognized fallback paths.
  • altimate-base-error-surfacing.test.ts (7) — non-rate-limit inference-time failures reach authorizedFetch's caller cleanly: 5xx pass-through, timeout/abort propagation, raw connection failure, malformed JSON body, and chat-time 401.

All suites are fully hermetic — fetch is injected via the shared FakeGateway fixture (spyOn(globalThis, "fetch")), so there is no live gateway, no credentials, and no network access anywhere in this PR. Each suite runs in its own isolated XDG/home tree (isolateAltimateBaseHome) so credential stores, config, and cache never collide across files or touch a real user directory. This is picked up by CI automatically via the existing test/altimate/** path filters — no workflow change needed.

The centralized-arming fix (the load-bearing change in this PR): FreeTierCapability.issueArmer() is a process-global capability that throws on a second call in the same process (by design — it's the security property that makes Altimate Base's consent gate unforgeable, see src/altimate/free/capability.ts). Every one of these 5 new suites, plus the two pre-existing files (altimate-base.test.ts, altimate-base-harness-smoke.test.ts), independently called issueArmer() at module scope. bun test test/altimate/ loads multiple test files into one worker process, so this throws "Altimate Base consent armer already issued for this process" as soon as a second armer-calling file loads — reproducible with just the two pre-existing files, before any of these suites existed.

Fixed by adding a consented() helper to the shared harness (_fixtures/altimate-base-harness.ts) that lazily claims issueArmer() exactly once per process and caches the returned armer in a module-level singleton. Because bun caches modules per process, every suite file that imports consented() — regardless of load order or how many files load it — shares that one cached armer. All 7 armer-calling files (5 new + 2 existing) now go through this shared helper instead of claiming their own armer. This adds no way to reset, re-claim, or otherwise weaken the one-shot guarantee issueArmer() already enforces; it is purely a cache in front of the single legitimate call, so the underlying security property (only one in-process caller can ever obtain the ability to arm the production consent authority) is unchanged — the existing "unforgeable consent" test in altimate-base.test.ts (which asserts a second issueArmer()/issueRedeemer() call throws) still passes unmodified.

Live-prod smoke testing against a real gateway is intentionally out of scope here and tracked separately / non-blocking for this PR — everything in this PR is fetch-injected and hermetic.

This PR stacks on #1199 (Altimate Base hosted model release) — the base is codex/altimate-base-release-final so the diff here shows only the harness + tests, not #1199's feature changes. It should be retargeted to main once #1199 merges.

How did you verify your code works?

Ran the full test/altimate/ directory in one bun test invocation, matching CI's own invocation and timeout (bun test --timeout 90000, from packages/opencode, per .github/workflows/ci.yml's typescript job):

bun test --timeout 90000 test/altimate/
 5103 pass
 653 skip
 0 fail
Ran 5756 tests across 186 files. [~75s]

Zero "consent armer already issued" errors — this is the actual acceptance bar, not file-by-file green (each file was also independently confirmed green: 11 + 9 + 5 + 21 + 7 = 53 new tests, all passing).

Also verified:

  • bun run typecheck (bun turbo typecheck) — clean, 13/13 tasks successful.
  • bun run script/upstream/analyze.ts --markers --base origin/main --strict — clean, no unmarked upstream-shared changes.
  • bunx prettier --check on all changed files — clean.

Screenshots / recordings

N/A — test-only change, no UI.

Checklist

  • I have tested my changes locally
  • I have not included unrelated changes in this PR

[!NOTE]
Low Risk
Test-only changes with no runtime behavior modifications; the shared consented() helper preserves the one-shot issueArmer() security model while fixing multi-file test loading.

Overview
Adds a hermetic Altimate Base e2e test layer built on shared _fixtures/altimate-base-harness.ts (isolated XDG home, gateway env reset) and _fixtures/fake-gateway.ts (spyOn(globalThis, "fetch") for /register and chat completions with scripted error modes). Five new suites cover registration failure gaps, provider catalog/defaultModel() behavior, register→Provider.list()→authorizedFetch inference, full describeRateLimit / describeRequestTooLarge branches, and inference-time 5xx/timeout/network/malformed-JSON/401 surfacing; a small harness smoke file exercises the fixtures.

Load-bearing harness fix: all suites (and the existing altimate-base.test.ts) now mint consent via shared consented(), which lazily claims FreeTierCapability.issueArmer() once per process so bun test test/altimate/ does not crash when multiple files load in one worker.

Also adds docs/internal/2026-09-04-altimate-base-e2e-harness-plan.md as the design contract. The planned context-clamp suite is not in this diff (still flagged in the doc). No production source changes; tests only, no CI workflow edits.

Reviewed by Cursor Bugbot for commit d2973f4ec904598eb2658e9c1a6eb7a72a264cc9. Bugbot is set up for automated code reviews on this repo. Configure here.

Summary by cubic

Adds 53 hermetic e2e tests across 5 suites for Altimate Base's free-tier client (closes #1247), and centralizes the process-global issueArmer() capability behind a shared consented() helper so all test files run together in one worker process without the "consent armer already issued" crash.

Coverage

  • New suites cover registration-failure mapping, catalog/provider isolation, register-to-inference round trips, rate-limit/budget message mapping, and inference-time error surfacing.
  • All tests inject fetch via the shared FakeGateway fixture with isolated XDG/home trees — no network, credentials, or live gateway.
  • Verified in one bun test invocation at CI's timeout (5103 pass, 0 fail), and the diff is test-only since it stacks on #1199 — retarget to main once that merges.

Consent armer

  • issueArmer() throws on a second in-process call by design; 7 files each claiming it at module scope crashed under bun's single-worker test loading.
  • consented() lazily claims the armer once per process and caches it, so all 7 files share it — the unforgeable-consent guarantee is unchanged and still covered by an existing test.