feat(ai): capture model calls with Rudimint

Merged
UR
4d ago
+76-11

ThinkEx's model calls aren't recorded anywhere we can evaluate or train on: PostHog keeps metadata, and content only with analytics consent. This routes every model call through Rudimint, which forwards it to Vercel AI Gateway unchanged and keeps a copy of the request and response.

How it works

flowchart LR
    TX[ThinkEx Worker] -->|same AI SDK request + x-rudimint-* headers| R[api.rudimint.com/vercel]
    R -->|forwarded unchanged| V[Vercel AI Gateway]
    V --> P[OpenAI / Anthropic / Google]
    P --> V --> R -->|streamed back| TX
    R -.->|copy after the response| S[(Rudimint)]
  • One place. Every call already builds its model in getWorkspaceAiLanguageModelForGatewayModel (src/features/workspaces/ai/gateway.ts). When RUDIMINT_KEY is set, createGateway gets Rudimint's base URL and headers; without it, calls go to Vercel exactly as before. Vercel's key, BYOK routing, fallbacks and streaming are untouched.
  • Tags. Each call says what it is:
Calltasksessionusermetadata
Chat turnchatthread iduser idworkspace, turn trace id, model
Compactionchat-compactionthread iduser idsame
Thread titlechat-titlethread iduser idsame
Image to markdownimage-to-markdown——workspace, item

Evals and the probe script stay untagged.

Rollout

  • Add RUDIMINT_KEY (a key from the ThinkEx tenant in Rudimint) to Infisical for each environment. It's optional on purpose: merging before the key exists changes nothing.
  • Content is captured for every user, regardless of the analytics cookie choice. ThinkEx's privacy policy should list Rudimint as a processor alongside Vercel and PostHog.

Validation

  • Typecheck, lint and the AI tests (51) pass.
  • Not yet exercised against the live Rudimint gateway; the first calls after the key is set will show up in Rudimint's admin under the ThinkEx tenant.

View with [code]smith Autofix with [code]smith
Need help on this PR? Tag @codesmith-bot with what you need. Autofix is disabled.

Summary by CodeRabbit

  • Improvements
    • AI chat replies, title generation, conversation compaction, and image-to-markdown processing can include task and workspace context in request tracking when configured.
    • Chat requests can also be associated with a shared session and user for easier monitoring.
  • Documentation
    • The privacy policy now explains that AI requests and responses pass through Rudimint, which retains copies for model evaluation and improvement.