feat(pipeline): Jev classifier front-runner for typed-decision calls
Summary
Ports the Jev integration (merged on argus-private as #301): TypeSafe Jev (jev-1.13.0, System One eval API) front-runs Argus's typed classification calls, cutting LLM spend on the pipeline's highest-volume classifier decisions. LLM paths stay untouched as the fallback — Jev only decides when its probabilities clear conservative thresholds.
backend/internal/jev — minimal client for POST /v1/systemone: batched questions over one state, pinned model, input-only cost math, jev.call.* telemetry mirroring llm.call.*, 120KB state cap, probability range validation.
Four cascades + one shadow, all fail-safe:
| Stage | Policy |
|---|---|
| Addressed judge (auto-resolve on push) | p≥0.97 addressed + p≥0.5 visible evidence → resolve; p≤0.05 → keep open; else → LLM judge |
| Convention-relation classifier (memory write gate) | 4-way choice per neighbor, all ≥0.95 → skip LLM; any uncertainty escalates whole batch |
| Intent verification | delivers + per-criterion + per-finding-scope nouls; confident all-clear only |
| Scoring FP pre-filter | fp≥0.97 AND defect≤0.5 → leaves judge prompt, re-enters as score-10 synthetic group |
| Triage (shadow only) | Parallel eval on ≤40-file diffs; agreement logged as jev.shadow.triage, NEVER routed |
BYOK credentials: per-installation keys via the encrypted provider_keys store (provider typesafe, repo-level then org-level) — Settings → Providers → TypeSafe Jev card beside Embeddings (masked key, replace/delete, optional custom endpoint). A stored key IS the consent — no flag needed. The env TYPESAFE_API_KEY path still requires jev_classifier opt-in (default OFF). Resolution is lazy — candidate-free pushes pay zero lookups.
Token accounting: Jev spend merges into each stage bucket (AutoResolve, Intent, Scoring, Conventions, Triage); previously-unbilled convention classifier spend is now billed on both legs.
Data egress: sanitized finding text, diff hunks, PR metadata/body, intent, stored conventions → api.typesafe.ai (or stored endpoint), only per the rules above. Documented in docs/self-hosting.md + backend/.env.example.
Test plan
-
go build ./... && go vet ./... && go test ./...— all packages green -
pnpm lint && pnpm typecheck && pnpm build— green - Tests: wire format, cascade thresholds, BYOK precedence (repo > org > env+flag), lazy resolution, custom endpoint, opt-out, bounded shadow join, spend accounting
- Adversarial-verify gate
- Shadow-mode data review before promoting triage
Summary by cubic
Adds TypeSafe's Jev classifier as a front-runner for addressed-judge, convention-relation, intent-verification, and scoring false-positive decisions, plus an observe-only triage shadow. Previously every decision went to the LLM; now confident Jev answers skip it, anything uncertain, missing, oversized, or failed falls back unchanged, and a flaky cross-PR test is fixed by capturing the lookup count before review-completion fanout.
New Features
- Addressed judge resolves at
p>=0.97with visible evidence, keeps open atp<=0.05, and escalates the middle band to the LLM. - Convention relations skip the LLM only when every neighbor's 4-way choice is
>=0.95; any uncertain answer escalates the whole batch. - Intent verification returns a verdict only on a confident all-clear across
delivers, acceptance criteria, and per-finding scope. - Scoring drops findings from the judge prompt only at
fp>=0.97plusdefect<=0.5, then re-enters them as score-10 synthetic groups for the audit trail. - The triage shadow logs
jev.shadow.triageagreement on up to 40 files; slow evals are abandoned without billing. - Jev receives only sanitized, delimiter-wrapped text; convention text that would truncate escalates to the LLM, and oversized-state evals emit
jev.call.failed. - Manual
model_pricingrows still win; models without one now use OpenRouter's public catalog, refreshed in the background, instead of costing $0.
Migration / Ops
- Enable per installation with a stored
typesafeprovider key (repo-level then org-level) or withTYPESAFE_API_KEYplus thejev_classifierfeature flag; no key means no Jev calls. - A stored key counts as consent to egress; the env-var path requires the flag because the shared key alone never sends tenant data.
- Stored
typesafeendpoints are validated and normalized on save; malformed URLs return a 400, and an invalid stored URL disables Jev instead of rerouting traffic. - Jev spend merges into existing stage token buckets; convention-relation spend is now billed on both legs.
- Mixed buckets keep a per-leg
Auxledger, andstats/modelsreports per-model totals for every scalar stage. - Egress is limited to opted-in installs: sanitized finding text, PR metadata/body, intent, diff hunks, and stored conventions.
- The OpenRouter catalog fallback is a public async fetch; disable it with
OPENROUTER_PRICING_ENABLED=falsein restricted-egress deployments.