feat(ai): capture model calls with Rudimint
Merged
UR
4d ago+76+76+01234567890123456789-11-11-01234567890123456789
ThinkEx's model calls aren't recorded anywhere we can evaluate or train on: PostHog keeps metadata, and content only with analytics consent. This routes every model call through Rudimint, which forwards it to Vercel AI Gateway unchanged and keeps a copy of the request and response.
How it works
flowchart LR
TX[ThinkEx Worker] -->|same AI SDK request + x-rudimint-* headers| R[api.rudimint.com/vercel]
R -->|forwarded unchanged| V[Vercel AI Gateway]
V --> P[OpenAI / Anthropic / Google]
P --> V --> R -->|streamed back| TX
R -.->|copy after the response| S[(Rudimint)]
- One place. Every call already builds its model in
getWorkspaceAiLanguageModelForGatewayModel(src/features/workspaces/ai/gateway.ts). WhenRUDIMINT_KEYis set,createGatewaygets Rudimint's base URL and headers; without it, calls go to Vercel exactly as before. Vercel's key, BYOK routing, fallbacks and streaming are untouched. - Tags. Each call says what it is:
| Call | task | session | user | metadata |
|---|---|---|---|---|
| Chat turn | chat | thread id | user id | workspace, turn trace id, model |
| Compaction | chat-compaction | thread id | user id | same |
| Thread title | chat-title | thread id | user id | same |
| Image to markdown | image-to-markdown | — | — | workspace, item |
Evals and the probe script stay untagged.
Rollout
- Add
RUDIMINT_KEY(a key from the ThinkEx tenant in Rudimint) to Infisical for each environment. It's optional on purpose: merging before the key exists changes nothing. - Content is captured for every user, regardless of the analytics cookie choice. ThinkEx's privacy policy should list Rudimint as a processor alongside Vercel and PostHog.
Validation
- Typecheck, lint and the AI tests (51) pass.
- Not yet exercised against the live Rudimint gateway; the first calls after the key is set will show up in Rudimint's admin under the ThinkEx tenant.
Need help on this PR? Tag @codesmith-bot with what you need. Autofix is disabled.
Summary by CodeRabbit
- Improvements
- AI chat replies, title generation, conversation compaction, and image-to-markdown processing can include task and workspace context in request tracking when configured.
- Chat requests can also be associated with a shared session and user for easier monitoring.
- Documentation
- The privacy policy now explains that AI requests and responses pass through Rudimint, which retains copies for model evaluation and improvement.