#48 [Feat]: Agent Evaluation & A/B Platform

Closes #16

Problem

The repo shipped with a chat-only experience — every assistant turn left the system as soon as the user clicked away. There was no signal for "is this prompt version better than that one," no way to audit a specific turn's reasoning, and no admin surface to seed a benchmark dataset or weight a rubric criterion.

What changed

Core eval platform (lib/eval/ + backend/agent/eval-agent.ts):

• 8 new Drizzle tables (prompt_template, prompt_variant, prompt_variant_assignment, eval_run, eval_feedback, eval_rubric, eval_benchmark, eval_judgment) on migrations 0017–0021.
• Standalone evalAgent graph (backend/agent/eval-agent.ts:graph, registered as evalAgent in langgraph.json) with mode="judge" / mode="benchmark" dispatched via routeFromInput conditional edges.
• EvalCallbackHandler (backend/eval/callback.ts, singleton in backend/callbacks.ts) writes eval_run on every chat-model handleLLMEnd, keyed by parent_message_id. Reads user id from run-level metadata.user_id (NOT configurable.userId).
• seedInitialPrompts (idempotent) populates 11 agent prompt templates, 11 rubrics (incl. judgeByLLM: criteria_alignment 0.4 / reasoning_clarity 0.4 / calibration 0.2), 1 generic rubric_default, and ≥1 benchmark row per agent.
• threads.kind enum split: chat / kb / eval-judge / eval-benchmark. The two eval-* kinds share the eval-agent graph but the run is fundamentally different — judge scores an existing production run, benchmark generates one from a prompt. Online Executions surfaces judge work (eval-judge); Benchmark Datasets is the only home for synthetic benchmarks (eval-benchmark).

API surface (app/api/eval/*):

• POST /api/eval/feedback — chat thumbs-up; resolves resp_... against eval_run.parent_message_id; silent no-op on no match (no dangling FK violation).
• GET / POST / PUT / DELETE /api/eval/prompts — template + variant lifecycle (create_template / create_cohort_variant / delete_cohort_variant / create_variant / update_variant_weight / batch_update_weights); system templates are 403 on DELETE.
• GET / POST /api/eval/rubrics — list + upsert by id.
• GET / POST /api/eval/benchmarks — per-agent benchmark CRUD + update_rubric action.
• POST /api/eval/benchmarks/run — Run Evaluate end-to-end.
• POST /api/eval/judge — Run Judge against an existing eval_run.
• GET /api/eval/runs/compare — A/B stats + first page of recent runs per agent.
• GET /api/eval/runs/page — per-agent paginated Online Executions (cursor-stable via PG row-value (created_at, id)).
• GET /api/eval/runs/[id] — single-run drilldown (run + feedback + judgment + first 20 spans).
• GET / POST /api/eval/assignments — sticky (userId, agent) bindings (cohort + single-agent reassignment).

Admin Eval Studio (app/admin/eval/*): per-agent cohort cards, unified <table className="table-fixed"> for Online Executions + Benchmark Datasets, Rubric weight editor with weighted-score badge, Prompt Studio card with system-prompt edit + A/B weight sliders.

Background agent rename: background_agent graph id → backgroundAgent (file slug unchanged; langgraph.json + every dispatch site updated together).

Landing page (/): Composable tools bento card → Evaluation & A/B (ScaleIcon, fuchsia hue). New 7th How-it-works row Evaluation below Knowledge base with its own motion demo (evaluation-explainer-demo.tsx) — direction is reverse of the KB explainer (run bubbles left, judge card right, criteria light up right-to-left, scores stamp back).

LLM-facing docs: public/llms.txt + public/llms-full.txt updated with eval platform, full /api/eval/* surface, 11 seeded agent IDs, threads.kind discriminator, evalAgent mode dispatch.

Tests

• 1406/1406 passing on 3da1f98.
• 33 new test cases across tests/api/eval/* (benchmarks-run, feedback, runs-page), tests/backend/agent/eval-agent.test.ts, tests/lib/eval/queries.test.ts, tests/frontend/landing/sections.test.tsx.

Docs

• docs/EVALUATION.md — full architecture + ops writeup (kind discriminator, agent IDs, seed contents, route map, no-op semantics).
• docs/APIS.md — eval section rewritten with kind-discriminator table, per-action prompt lifecycle body, benchmark row denormalization, judge/benchmark kind semantics.
• docs/DEPLOY.md — eval tables + kind enum migration note.
• docs/DB.md — threads.kind row updated to 4 values; Code → table map gets all 8 eval tables.
• docs/OBSERVABILITY.md — sidebar filter rationale updated to mention the two eval-* kinds.
• docs/LANDING.md — section list updated to 7 explainer rows + Evaluation & A/B bento card.
• README.md — Features bullet for Agent Evaluation Studio; file tree + Database section updated.
• public/llms.txt, public/llms-full.txt — eval platform surfaced.

🤖 Generated with Claude Code

Summary by cubic

Adds an Agent Evaluation & A/B platform with prompt templates/variants, sticky traffic splits, an LLM judge, and an Admin Studio for benchmarks and online executions — gated by admin role with strict data ownership. Closes #16.

• New Features

  • Core eval pipeline: 8 tables and an evalAgent graph with mode="judge" and mode="benchmark".
  • Prompt A/B: agents load prompts via getAgentPrompt; EvalCallbackHandler records eval_run with templateId/variantId, keyed by parent_message_id.
  • Seeded 11 agent prompt templates, per-agent rubrics, and rubric_default; sticky traffic splits and user-cohort assignments.
  • APIs: /api/eval/feedback, /api/eval/prompts, /api/eval/rubrics, /api/eval/benchmarks (+ /run), /api/eval/judge, /api/eval/runs/compare, /api/eval/runs/page, /api/eval/runs/[id], /api/eval/assignments.
  • Admin Eval Studio: Executions (per-agent pagination + traces), Benchmarks (latest judgment snapshot), Prompt Studio (system prompts + traffic split), Rubric editor, User Assignments; chat UI adds thumbs up/down.
  • threads.kind now includes eval-judge and eval-benchmark; Online Executions exclude only eval-benchmark.
  • Background graph id renamed to backgroundAgent; docs and landing updated.

• Bug Fixes

  • Secured eval APIs: all mutations require role="admin"; runs pagination/compare are admin-only; /api/eval/runs/[id] is owner-or-admin (non-owners get 404).
  • EvalCallbackHandler skips writes when threadId or userId is missing to avoid FK errors.
  • KB OCR now honors managed prompts: page-to-markdown passes userId to getAgentPrompt, so Prompt Studio edits affect ingestion.

Greptile Summary

Adds a database-backed agent evaluation and A/B-testing platform.

• Introduces prompt variants, sticky assignments, execution capture, feedback, rubrics, benchmarks, and LLM judgments.
• Adds authenticated evaluation APIs and an administrative Eval Studio.
• Registers a dedicated evalAgent graph and adds evaluation controls to chat and landing-page surfaces.
• Adds migrations, documentation, and focused evaluation tests.

Confidence Score: 1/5

The PR is not safe to merge until Eval Studio authorization, prompt attribution, benchmark lifecycle persistence, and page-to-markdown prompt loading are corrected.

Non-admin users can access and mutate globally scoped evaluation resources, normal executions can be attributed to the wrong prompt variant, failed benchmark attempts retain stale success data, and synthetic benchmark threads accumulate without cleanup.

Files Needing Attention: app/api/eval/**, lib/eval/callback.ts, lib/eval/queries.ts, backend/agent/eval-agent.ts, backend/node/kb/page-to-markdown-node.ts

Important Files Changed

Flowchart

%%{init: {'theme': 'neutral'}}%%
flowchart LR
  U[Chat or benchmark input] --> A[Agent prompt loader]
  A --> V[Sticky prompt variant]
  V --> L[LLM execution]
  L --> C[Eval callback]
  C --> R[(eval_run)]
  R --> J[evalAgent judge]
  J --> S[(eval_judgment)]
  J --> B[(Benchmark latest result)]
  R --> D[Admin Eval Studio]
  S --> D
  B --> D

Fix the following 5 code review issues. Work through them one at a time, proposing concise fixes.

---

### Issue 1 of 5
app/api/eval/prompts/route.ts:27
**Eval APIs omit admin authorization**

Any authenticated non-admin can call these routes directly because they use bare `withAuth`, bypassing the admin check enforced only by the page. This exposes cross-user evaluation data and permits unauthorized prompt, traffic-weight, rubric, benchmark, and assignment mutations. **How this was verified:** The eval handlers use session-only authentication, while the established admin handlers use the admin-role overload and the run queries contain no user ownership filter.

### Issue 2 of 5
lib/eval/callback.ts:58-62
**Executed variants are misattributed**

When a user receives a non-first A/B variant, the agent consumes its prompt without propagating the returned variant ID into callback metadata. The callback therefore falls back to the first enabled variant, causing the run and resulting A/B statistics to be attributed to a prompt that was not executed.

### Issue 3 of 5
backend/agent/eval-agent.ts:396-400
**Failed benchmarks retain stale results**

When a benchmark has a previous successful result and a later judge invocation throws, this branch returns failed graph state without updating the benchmark's denormalized latest-result fields. The Benchmark Datasets table consequently continues presenting the previous score and success status as the latest result.

### Issue 4 of 5
backend/node/kb/page-to-markdown-node.ts:44
**Managed OCR prompt is bypassed**

Every page-to-markdown invocation omits the user ID, so `getAgentPrompt` immediately returns the source fallback instead of resolving a database variant. Administrators can edit and deploy the seeded `pageToMarkdown` prompt in Prompt Studio, but those changes never affect KB ingestion.

### Issue 5 of 5
backend/agent/eval-agent.ts:170-178
**Benchmark threads are never cleaned up**

Each benchmark execution creates an `eval-benchmark` thread here, but the compiled graph has no cleanup node or deletion edge after judging. Repeated evaluations therefore leave permanent synthetic threads and their associated evaluation and observability records, causing unbounded database accumulation.

_{Reviews (1): Last reviewed commit: "style(landing): polish Evaluation explai..." | Re-trigger Greptile}

Greptile also left 5 inline comments on this PR.

Context used:

• Context used - CLAUDE.md (source)