feat: verifier-gated model router (run cheap → verify → escalate)

Draft
AN
2mo ago
+2317-38

What does this PR do?

Adds a verifier-gated model router: run a cheap model first, verify the result deterministically (dbt build), and escalate to a stronger model only when verification fails. Flag-gated via ALTIMATE_ROUTER (default off) — the normal single-model run path is byte-for-byte unchanged.

  • packages/opencode/src/router/verifier.ts — deterministic Verdict from dbt build/dbt test. Spoof-proof (parses the last summary + uses exitCode as backstop, so a model-emitted fake Done. PASS=… ERROR=0 cannot pass the gate). Pluggable Impl; honest unverifiable state when a gate can't run.
  • packages/opencode/src/router/router.ts — the escalation ladder. Runs each tier, verifies, escalates on a failed verdict with the exact failing checks as context, stops at the first pass. Per-tier exception handling (a transient tier error escalates instead of aborting). Ladder overridable via ALTIMATE_ROUTER_LADDER.
  • packages/opencode/src/router/policy.ts — where the ladder comes from: a static default, or a per-context policy fetched from the altimate API when ALTIMATE_API_KEY is set (with AbortSignal timeouts + sanitizeTiers validation/capping; degrades to static on any failure).
  • packages/opencode/src/router/verdict.ts — a machine-checkable verdict envelope (schemaVersion, per-attempt history, checks, evidence hash, optional signature).
  • packages/opencode/src/cli/cmd/run.ts — orchestration when the flag is on. Only routes verifiable (dbt) workspaces; a non-dbt project runs once with the user's own model (no silent downgrade). dbt build runs with a hard timeout.

The customer routing policy endpoint lives in altimate-backend (separate PR). This PR ships the client + the static default.

Type of change

  • New feature (non-breaking change which adds functionality)

Issue for this PR

Closes #853

How did you verify your code works?

  • 51 unit tests (test/router/*.test.ts), including adversarial cases: dbt summary-line injection, ANSI/huge/multi-summary output, endpoint response validation + cost-bomb capping, and per-tier exception escalation.
  • 3 env-gated E2E suites (test/router/*.e2e.test.ts) run with real dependencies (no mocks): real dbt build in a container (incl. a spoof-attack that fails to fool the gate), real OpenRouter model calls with real escalation, and real-network policy fallback + adversarial endpoint responses.
  • tsgo --noEmit typecheck clean on all changed files; marker check clean.
  • Live CLI smoke (ALTIMATE_ROUTER=1 on a real dbt project): routed to the cheap tier → real agent run → real dbt build verify → verdict emitted, no escalation needed.
  • Reviewed via multi-model consensus (/consensus:code-review); all CRITICAL/MAJOR findings (non-dbt gating, external-call timeouts, per-tier exception handling, honest unverifiable verdict, endpoint hardening) applied before opening.

Checklist

  • My code follows the style guidelines of this project
  • I have performed a self-review of my own code
  • I have commented my code, particularly in hard-to-understand areas
  • I have added tests that prove my fix is effective or that my feature works
  • New and existing unit tests pass locally with my changes
  • Draft — execute()-level follow-ups (reuse one session across tiers; finally cleanup of per-tier listeners/tracer) tracked for a follow-up before marking ready

Summary by cubic

Adds a verifier‑gated model router: run a cheap model, verify with dbt build, and escalate only on failure. Optional, flag‑gated equivalence verifier compares base↔head compiled SQL and falls back to build/test; escalated tiers reuse the same session; default off.

  • New Features

    • Router: decision‑aware escalation (escalate on FAILED/ PROVEN_DIFFERENT, never on UNDECIDABLE); stops at first pass; reuses the initial session across tiers.
    • Verifier: spoof‑proof dbt parsing (last summary + exitCode backstop) with a 300s timeout; verdicts include strength (UNVERIFIABLE < BUILD < DBT_TEST < EQUIVALENCE) and decision (OK | PROVEN_DIFFERENT | UNDECIDABLE | FAILED).
    • Equivalence: behind ALTIMATE_ROUTER_EQUIVALENCE=1 using a git+dbt ReferenceResolver to produce base↔head compiled SQL pairs and altimate_core.equivalence; on UNDECIDABLE or resolver/engine errors, falls back to build/test without silent passes.
    • Verdict envelope: v2 schema with strength/decision, attempt history, checks, evidence hash, optional signature.
    • Policy/CLI: static ladder or Altimate API policy (ALTIMATE_API_KEY/ALTIMATE_API_URL) with outcome reporting; routes only in verifiable dbt workspaces; non‑dbt runs once; env: ALTIMATE_ROUTER, ALTIMATE_ROUTER_EQUIVALENCE, ALTIMATE_ROUTER_LADDER, ALTIMATE_API_KEY, ALTIMATE_API_URL.

  • Bug Fixes

    • CLI always removes per‑run signal handlers and finalizes the tracer in a try/finally, preventing leaks when a tier throws.
    • E2E router tests skip (not fail) when E2E_IMG or OPENROUTER_API_KEY are missing.
    • E2E cleanup removes the redundant sudo rm -rf and relies on rmSync, avoiding unnecessary privileges.

Summary by CodeRabbit

  • New Features

    • Verifier-gated router orchestration in the run command enables multi-tier policy routing with workspace verification.
    • Integrated dbt-based workspace verification to validate outcomes deterministically.
    • Escalation logic threads failure context across tiers to guide subsequent attempts.
    • Verdict envelopes capture and report routing outcomes.
    • Support for both API-driven and static routing policies.

  • Documentation

    • Added comprehensive router module documentation.

  • Tests

    • Added extensive unit and end-to-end test coverage for router components.