fix: build release binaries against the live models.dev catalog

Merged
AN
2mo ago
+595-6

Issue for this PR

Closes #1186

Type of change

  • Bug fix
  • New feature
  • Refactor / code improvement
  • Documentation

What does this PR do?

.github/workflows/release.yml built every platform binary with MODELS_DEV_API_JSON: test/tool/fixtures/models-api.json. build.ts does process.chdir(packages/opencode) first, so that resolved to the checked-in test fixture, which was written over src/provider/models-snapshot.ts and compiled into the binary. Shipped binaries therefore embed a test fixture whose newest entry is 2026-03-30 as their bundled models.dev catalog.

This PR removes that one env line from the release workflow, and adds a validation guard to build.ts because removing it makes the release build depend on models.dev.

Two properties the guard exists to hold:

  1. A failed, slow, or unreachable models.dev fails the build loudly. It never falls back to a stale or partial catalog.
  2. Validation is not satisfied by a provider merely appearing — the major providers must carry a non-empty models object.

Why a guard at all. fetch resolves for 4xx/5xx rather than throwing, so without a check a load-balancer error page flows straight into the snapshot. An HTML body would at least break the build at parse time, but a JSON error body such as {"error":"bad gateway"} is valid TypeScript and would have shipped as a catalog with no providers in it.

The guard now:

  • bounds the fetch with AbortSignal.timeout(60s), plus a setTimeout backstop at 75s that exits non-zero. The backstop is not redundant — AbortSignal.timeout cannot cancel a blocked getaddrinfo() (src/provider/models.ts, #1052 D14), and a request can hang unresolved with the abort firing but the promise never settling. Its limit, stated honestly after review: it is a timer on the event loop, so it cannot preempt a genuinely blocked main thread either; in that case the workflow timeout-minutes remains the real backstop. It is given a 15s margin over the abort signal so it never shadows the precise per-stage messages
  • reads the body inside its own try, so a host that sends headers then stalls mid-body reports the URL and timeout instead of a bare AbortError
  • requires the payload to parse, be a non-empty provider object, and — in every mode — that every provider entry carries a string id and an object models, and that every model inside it is an object with a limit object carrying a numeric context. That last part is not cosmetic: Provider.fromModelsDevModel dereferences model.limit.context with no guard while every neighbouring field uses ?./??, and the runtime's own screen (isCatalogEntry) never looks inside the models map, so a limit-less record crashes provider initialisation on cold start. Deliberately not the zod Provider schema, which requires an options record real models.dev entries lack and would reject valid live data
  • in strict mode additionally requires at least 50 providers and that anthropic, openai and google are present with models
  • logs the accepted counts (105 providers (anthropic=23, openai=47, google=35)) so a shrink is visible in the build log

Strict mode, and why it is scoped. OPENCODE_MODELS_URL and MODELS_DEV_API_JSON are legitimate ways to point a build at a small private catalog, and that worked before this PR; an unconditional 50-provider floor broke it. Strict is now:

const strictCatalog = !!process.env.OPENCODE_RELEASE || (!modelsFile && !modelsUrlOverride)

ON for every release build and for any plain default-endpoint build; OFF only for a non-release build with an explicit override. Keying on OPENCODE_RELEASE as well as the overrides is deliberate — pointing a release at a custom catalog must not be a way to skip the floor. Both properties above hold unconditionally for every shipped binary.

Risk this introduces, stated plainly. The release build now depends on models.dev being reachable. If it is down or unresolvable, the build fails and the release stops — intended, and what upstream already accepts (publish.yml's Build step sets only OPENCODE_VERSION, OPENCODE_RELEASE, GH_REPO, GH_TOKEN; no MODELS_DEV_API_JSON). A blocked release is recoverable by re-running the job; a silently stale binary shipped to users is not. Fetch-with-fallback-to-fixture would avoid the block but reintroduces shipping a stale catalog unnoticed, so it is not the default.

Known limitation, deliberately not fixed here. The eight matrix jobs each fetch independently, so a models.dev deploy landing mid-release could give different platform binaries slightly different catalogs. The snapshot is only a cold-start fallback the runtime replaces on first successful fetch, so this self-corrects rather than persisting. Fixing it properly means a fetch-once job passing the catalog as an artifact — a larger change deserving its own review, and upstream has the same property. Raised by three reviewers; answered in-thread.

Things I checked before assuming this was safe to remove:

  • Does it break tests? No. The fixture is consumed by tests via direct file paths (test/preload.ts sets OPENCODE_MODELS_PATH; read.test.ts / truncation.test.ts / llm.test.ts / llm-native-recorded.test.ts open it by path). None read the MODELS_DEV_API_JSON env var. The fixture file is untouched by this PR.
  • Does anything else depend on the pin? ci.yml:506 and packages/opencode/script/pre-release-check.ts:76 also set it. Both are left alone — a hermetic build with no network is correct for CI. (ci.yml also sets OPENCODE_RELEASE: "1", so its sanity build runs strict against the real fixture, which passes.)
  • Does any reproducible-build property depend on it? No. build-inputs.json is a per-build staleness stamp written from the files on disk after the snapshot is generated, so it stays self-consistent. Nothing in CI rebuilds and compares stamps across runs. Its consumer (test/install/smoke-test-binary.test.ts) degrades to test.skip on a mismatch rather than failing, and returns early when dist/ is absent — the case in the release test job. The one real consequence is that rebuilding the same tag later produces a different stamp, since the catalog moves.
  • Is build.ts upstream-shared? It is listed in script/upstream/utils/config.ts under the Altimate-owned paths, so no altimate_change markers are required; the marker check passes.

How did you verify your code works?

Confirmed the bug end-to-end against a real shipped artefact. Ran the published 0.9.7 binary in an isolated HOME + XDG dirs with OPENCODE_MODELS_URL pointed at a closed port, so it served only its embedded snapshot:

EXIT=0  elapsed=2s   stdout: 91 models   stderr: 0 bytes   models.json cached: 0

Comparing that binary's openai ids against the fixture and the live catalog:

shipped-binary openai ids: 46      fixture: 47      live: 47
shipped - fixture: []                                  <- nothing in the binary that isn't in the fixture
fixture - shipped: ['gpt-5-chat-latest']               <- filtered elsewhere
shipped - live:   ['codex-mini-latest', 'gpt-5-codex', 'gpt-5.1-chat-latest', 'gpt-5.1-codex',
                   'gpt-5.1-codex-max', 'gpt-5.1-codex-mini', 'gpt-5.2-codex', 'o1-mini',
                   'o1-preview', 'o3-deep-research', 'o4-mini-deep-research']
live - shipped:   ['chatgpt-image-latest', 'gpt-5.5', 'gpt-5.5-pro', 'gpt-5.6', 'gpt-5.6-luna',
                   'gpt-5.6-sol', 'gpt-5.6-terra', 'gpt-image-1', 'gpt-image-1-mini',
                   'gpt-image-1.5', 'gpt-image-2', 'gpt-realtime-2.1']

So the shipped catalog is the fixture, it offers 11 ids the live catalog has dropped, and it is missing all of gpt-5.5/gpt-5.6. Provider counts: fixture 105, live 207.

Exercised the guard in both modes by running build.ts with MODELS_DEV_API_JSON pointed at crafted payloads. Rejections abort during early validation, and git status confirms models-snapshot.ts is untouched because the guard runs before the write.

payloadstrict (release)non-strict (explicit custom catalog)
{"error":"bad gateway"}has 1 malformed provider entries: errorsame
<html>502 Bad Gateway</html>is not valid JSONsame
[1,2,3]is not a provider objectsame
{}is emptyis empty
fixture with openai deletedis missing required providers: openaiaccepted (floor not applied)
80 keys, every value the string "junk"has 80 malformed provider entries: p0, …same
one malformed entry among 104 validhas 1 malformed provider entries: rogue (no string id)same
one model with limit deletedopenai/gpt-5.2-codex (no limit)same
one model value set to nullanthropic/claude-opus-4-5-… (not an object)same
every provider present, all models: {}has no usable models for: anthropic, openai, googleaccepted (floor not applied)
1-provider private cataloghas only 1 providers, expected at least 50exit 0 — the regression this fixes

Regression check on the env handling. A set-but-empty MODELS_DEV_API_JSON previously produced fetch(""):

before: error: models.dev fetch from  failed ...  code: "ERR_INVALID_URL"
after:  error: models.dev fetch from http://127.0.0.1:9/api.json failed or timed out after 60000ms

Happy path. The real fixture passes strict — 105 providers (anthropic=23, openai=47, google=35) then Generated models-snapshot.ts. The live catalog also passes (207 providers). The structural check is not over-tight: 0 of 105 fixture providers, 0 of 144 committed-snapshot providers and 0 of 207 live providers fail it, and at the model level 0 of 4108 fixture models, 0 of 5299 committed-snapshot models and 0 of 7487 live models fail it.

Gates. Run on this branch and, for comparison, on unmodified main:

gatemain (baseline)this branch
bun run typecheckpass (13/13)pass (13/13)
bun run script/upstream/analyze.ts --markers --base main --strictpasspass
bun run lint5870 warnings, 1 error5870 warnings, 1 error
bun test test/branding/{build-integrity,upstream-merge-guard}.test.ts test/plugin/codex-allowlist.test.ts—131 pass, 0 fail

The single lint error is pre-existing and unrelated — typescript-eslint(consistent-return) at packages/http-recorder/test/record-replay.test.ts:285, identical on unmodified main. Warning count is identical to baseline.

Not verified: I did not run an actual release build against live models.dev, so the network path is exercised only through the local-file path plus a live fetch of the same URL outside the build. I also did not simulate a genuinely blackholed DNS resolver; the backstop is reasoned from the #1052 D14 findings rather than reproduced, and as noted above it does not cover a fully blocked event loop. I did verify the timer ordering — a refused connection reports the precise fetch failed or timed out message, not the backstop's.

Screenshots / recordings

Not a UI change.

Checklist

  • I have tested my changes locally
  • I have not included unrelated changes in this PR

Summary by CodeRabbit

  • New Features
    • Release builds now use the latest live models catalog instead of a bundled test catalog.
  • Bug Fixes
    • Strengthened catalog validation to detect malformed or incomplete provider and model data.
    • Builds now fail clearly when the catalog cannot be fetched, read, or does not meet release requirements.
  • Reliability
    • Catalog diagnostics provide clearer build information without exposing sensitive connection details.
    • Custom catalog configurations continue to support operator-controlled validation behavior.

[!NOTE]
Medium Risk
Release artifacts and the release pipeline now depend on live models.dev reachability and response quality; a outage blocks shipping but avoids silently stale embedded catalogs.

Overview
Release builds no longer pin the models catalog to the CI test fixture, which had been baked into shipped binaries as a stale models-snapshot.ts. The release workflow drops MODELS_DEV_API_JSON so build.ts fetches models.dev at build time (same as upstream publish behavior).

build.ts now treats catalog loading as a hard gate: timed fetch with HTTP/res.ok checks, optional local file via MODELS_DEV_API_JSON, and assertUsableCatalog (new models-catalog.ts) so error pages or {"error":…} JSON cannot ship as an empty catalog. Strict validation applies when OPENCODE_RELEASE is set or the default URL is used (≥50 providers, anthropic/openai/google with models); custom catalog overrides stay non-strict. Build logs get a safe origin-only diagnostic and provider counts.

Tests in build-models-catalog.test.ts cover validation modes, malformed entries, and that fetch failures do not leak URL secrets.

Reviewed by Cursor Bugbot for commit b044ab0e64db9834489797448a2e5ac29b261189. Bugbot is set up for automated code reviews on this repo. Configure here.