feat(pipeline): Jev classifier front-runner for typed-decision calls

Merged
BE
2mo ago
+4254-256

Summary

Ports the Jev integration (merged on argus-private as #301): TypeSafe Jev (jev-1.13.0, System One eval API) front-runs Argus's typed classification calls, cutting LLM spend on the pipeline's highest-volume classifier decisions. LLM paths stay untouched as the fallback — Jev only decides when its probabilities clear conservative thresholds.

backend/internal/jev — minimal client for POST /v1/systemone: batched questions over one state, pinned model, input-only cost math, jev.call.* telemetry mirroring llm.call.*, 120KB state cap, probability range validation.

Four cascades + one shadow, all fail-safe:

StagePolicy
Addressed judge (auto-resolve on push)p≥0.97 addressed + p≥0.5 visible evidence → resolve; p≤0.05 → keep open; else → LLM judge
Convention-relation classifier (memory write gate)4-way choice per neighbor, all ≥0.95 → skip LLM; any uncertainty escalates whole batch
Intent verificationdelivers + per-criterion + per-finding-scope nouls; confident all-clear only
Scoring FP pre-filterfp≥0.97 AND defect≤0.5 → leaves judge prompt, re-enters as score-10 synthetic group
Triage (shadow only)Parallel eval on ≤40-file diffs; agreement logged as jev.shadow.triage, NEVER routed

BYOK credentials: per-installation keys via the encrypted provider_keys store (provider typesafe, repo-level then org-level) — Settings → Providers → TypeSafe Jev card beside Embeddings (masked key, replace/delete, optional custom endpoint). A stored key IS the consent — no flag needed. The env TYPESAFE_API_KEY path still requires jev_classifier opt-in (default OFF). Resolution is lazy — candidate-free pushes pay zero lookups.

Token accounting: Jev spend merges into each stage bucket (AutoResolve, Intent, Scoring, Conventions, Triage); previously-unbilled convention classifier spend is now billed on both legs.

Data egress: sanitized finding text, diff hunks, PR metadata/body, intent, stored conventions → api.typesafe.ai (or stored endpoint), only per the rules above. Documented in docs/self-hosting.md + backend/.env.example.

Test plan

  • go build ./... && go vet ./... && go test ./... — all packages green
  • pnpm lint && pnpm typecheck && pnpm build — green
  • Tests: wire format, cascade thresholds, BYOK precedence (repo > org > env+flag), lazy resolution, custom endpoint, opt-out, bounded shadow join, spend accounting
  • Adversarial-verify gate
  • Shadow-mode data review before promoting triage

Summary by cubic

Adds TypeSafe's Jev classifier as a front-runner for addressed-judge, convention-relation, intent-verification, and scoring false-positive decisions, plus an observe-only triage shadow. Previously every decision went to the LLM; now confident Jev answers skip it, anything uncertain, missing, oversized, or failed falls back unchanged, and a flaky cross-PR test is fixed by capturing the lookup count before review-completion fanout.

New Features

  • Addressed judge resolves at p>=0.97 with visible evidence, keeps open at p<=0.05, and escalates the middle band to the LLM.
  • Convention relations skip the LLM only when every neighbor's 4-way choice is >=0.95; any uncertain answer escalates the whole batch.
  • Intent verification returns a verdict only on a confident all-clear across delivers, acceptance criteria, and per-finding scope.
  • Scoring drops findings from the judge prompt only at fp>=0.97 plus defect<=0.5, then re-enters them as score-10 synthetic groups for the audit trail.
  • The triage shadow logs jev.shadow.triage agreement on up to 40 files; slow evals are abandoned without billing.
  • Jev receives only sanitized, delimiter-wrapped text; convention text that would truncate escalates to the LLM, and oversized-state evals emit jev.call.failed.
  • Manual model_pricing rows still win; models without one now use OpenRouter's public catalog, refreshed in the background, instead of costing $0.

Migration / Ops

  • Enable per installation with a stored typesafe provider key (repo-level then org-level) or with TYPESAFE_API_KEY plus the jev_classifier feature flag; no key means no Jev calls.
  • A stored key counts as consent to egress; the env-var path requires the flag because the shared key alone never sends tenant data.
  • Stored typesafe endpoints are validated and normalized on save; malformed URLs return a 400, and an invalid stored URL disables Jev instead of rerouting traffic.
  • Jev spend merges into existing stage token buckets; convention-relation spend is now billed on both legs.
  • Mixed buckets keep a per-leg Aux ledger, and stats/models reports per-model totals for every scalar stage.
  • Egress is limited to opted-in installs: sanitized finding text, PR metadata/body, intent, diff hunks, and stored conventions.
  • The OpenRouter catalog fallback is a public async fetch; disable it with OPENROUTER_PRICING_ENABLED=false in restricted-egress deployments.