[BREAKING_CHANGE] feat(usage): spend user credits on AI chat and web search before workspace credits (server)
Each user spends their own user credits on AI chat and its web searches before workspace credits are used. Nothing changes until AI_USER_CREDITS_LIMIT or a plan value is set. The front (meter, banners, fallback setting) comes in a follow-up PR.
How it works
Definition. User credits are a usage-limit quota default counted in the new UsageUnit.USER_CREDIT. AI declares one in USAGE_LIMIT_DEFINITIONS:
- operations
[AI_CHAT_TOKEN, WEB_SEARCH], sharing one counter (WEB_SEARCHis now AI chat search only); - one counter per user (
spenderType: 'userWorkspace',isPerSpender: true); - a UTC calendar week;
- the value from
AI_USER_CREDITS_LIMITin microCredits, whichUSAGE_LIMIT_DEFAULTS_BY_PLANcan set per plan. Unset or 0 means no user credits.
The engine is not AI specific: another resource gets user credits by declaring its own USER_CREDIT default. Several USER_CREDIT windows can apply at once, for example a day and a week. The user pays while every window has room.
Who pays. A charge that names a user and matches a USER_CREDIT quota on resource, operation and the userWorkspace spender is paid from user credits first.
- User credits pay for chat turns and titles, and chat web search.
- Workspace credits pay for:
- agent runs, workflow AI agent steps, generate-text and AI evaluation (
AI_WORKFLOW_TOKEN); - the web searches of agent runs and workflow AI agent steps (
AI_WORKFLOW_WEB_SEARCH), even when a user started them; - agent triggers, which name no user;
- app charges, logic functions and email, which have no
USER_CREDITdefault.
- agent runs, workflow AI agent steps, generate-text and AI evaluation (
One quota path. The user allowance is a third quota counter kind, next to limits and the workspace allowance. It is built by the same builder, warmed from the same ClickHouse consumption rows, debited by the same script and admitted through the same path. UsageLimitQuotaService owns all of it, as on main. Reading the quota list now warms its counters in Redis, as a debit does, instead of only reading them.
Debit. One WARM_AND_CONSUME_COUNTERS_SCRIPT call per charge.
- Keys are tagged
user(each user window, with its limit) orworkspaceCredits(the workspace allowance and workspace-wide CREDIT limits). - The script computes the user share as the smallest room left across the windows, capped at the amount. It debits every window by that share and the
workspaceCreditskeys by the rest, atomically. - If a tagged key is cold, no tagged key is debited until the retry has warmed them, so a share is never applied twice.
- If the warm fails, the charge is admitted as on main and recorded as user credits, in both fallback modes. User windows still in Redis take the full amount, workspace keys take none, and a cold counter catches up from the recorded row when it warms. The same applies when the whole debit fails, for example Redis down: the charge is recorded as user credits. Only a failure before the counters are known, when it is not even known whether user credits apply, records it as workspace credits as on main. Workspace credits are otherwise never spent because a user window could not be read. In fallback mode, a user already out of room goes over their window for that charge instead of spilling.
What blocks. findBlockingCounters is shared by debit and admission.
- While the user pays, the workspace allowance and workspace CREDIT limits do not block.
- A full user window blocks only in STOP mode.
When user credits run out. The billing-owned setting billingCustomer.userCreditsFallback decides:
WORKSPACE_CREDITS(the default): charges spill to workspace credits.STOP: the user's AI usage stops until the window renews (Monday 00:00 UTC for the AI default).
It is set with updateUserCreditsFallback (user sessions only, BILLING permission) and read from the cached currentBillingSubscription. The column comes with a 2.49 instance command.
Recording. usageEvent.userCreditsUsedMicro stores the user share, and creditsUsedMicro stays the total, so workspace credits are total minus user credits. computeRowConsumed holds the one rule:
USER_CREDITcounters countuserCreditsUsedMicro;- workspace CREDIT limits count
creditsUsedMicro - userCreditsUsedMicro.
The limit-form preview uses the same rule. Billing-period workspace and user credits come from one query.
AI chat.
- The step debit returns
{ stopReason, userCreditsUsedMicro, userAllowanceExhaustedUntil }. - STOP ends the turn after the step that crosses the window.
- The chat publishes
user-credits-exhausted(isFallingBackToWorkspaceCredits,renewsAt) from the debit that crosses a window, either a step or the end-of-turn web search.isFallingBackToWorkspaceCreditsis true only when nothing stops the turn: it is false in STOP mode and when the workspace allowance is exhausted by the same debit. - The turn records the summed user share.
- New sends in STOP mode are refused by the send preflight. A queued turn that starts afterwards runs one step, then stops, as main does for the workspace allowance.
- The workspace setup chat is admitted like a chat send of its creator (AI chat tokens, the creator as spender) instead of checking workspace credits only. A creator with user credits left can start it after workspace credits run out, and an exhausted AI usage limit now refuses it too.
API changes
BillingCustomer.userCreditsFallback, theUserCreditsFallbackenum and theupdateUserCreditsFallbackmutation.- On
BillingResourceCreditUsageandAdminPanelWorkspaceUsage, the workspace-only amounts are renamed:usedCredits,grantedCredits,rolloverCredits,totalGrantedCreditsandremainingCredits(admin panel only) becomeworkspaceCreditsUsed,workspaceCreditsGranted,workspaceCreditsExtraGranted,workspaceCreditsTotalGrantedandworkspaceCreditsRemaining.rolloverCreditswas the sum of every active credit grant, not only rollovers, henceworkspaceCreditsExtraGranted.userCreditsUsedis added. UsageBreakdownItemgetsuserCreditsUsedandworkspaceCreditsUsed, andUsageTimeSeriesgetsworkspaceCreditsUsed.creditsUsedstays the total on both.- Every credits query in
UsageAnalyticsServicecomputes workspace credits in SQL and still returns the total. It subtracts the summed aliases (creditsUsedMicro - userCreditsUsedMicro): ClickHouse binds a column name to a same-named alias, sosum(creditsUsedMicro) - sum(userCreditsUsedMicro)next tosum(creditsUsedMicro) AS creditsUsedMicrofails as a nested aggregate. The usage page shows workspace credits only, and breakdowns are ordered by them. The admin AI usage by workspace keeps showing the total, since it tracks AI cost. UsageOperationType.AI_WORKFLOW_WEB_SEARCHfor the web searches of agent runs, so automation searches stay on workspace credits, like their tokens. They keep the user who started them, so per-member limits and usage still count them. Admins can limit it likeWEB_SEARCH. An existingWEB_SEARCHlimit no longer counts agent-run searches; limits on all AI operations still do. Prod has twoWEB_SEARCHlimits, both workspace-wide, on workspaces with no agent runs and no web search in the last 60 days.UsageUnit.USER_CREDIT. It is not in anyallowedUnits, so admins cannot createUSER_CREDITusage limit rows yet.- The exhausted kind
allowancebecomesworkspaceAllowance, anduserAllowanceis added. Front checks are updated. The workspace allowance Redis key string is unchanged, so live counters stay warm. - New subscription event
user-credits-exhausted. - The removed
BillingResourceCreditUsagefields fail the breaking-changes check; the[BREAKING_CHANGE]title tag accepts them.
Vocabulary
- Credits are workspace credits or user credits. Their counters are the workspace allowance and the user allowance.
- Plain "credits" only names a value holding both.
Rollout constraints
- Migration 014 must run before this code deploys. Every usage insert now writes
userCreditsUsedMicro, and the readers use it. ClickHouse migrations run after code deploys, so 014 has to land first on its own, or be run by hand.- 014 also rebuilds the
consumption_by_scopeandbilling_by_workspace_periodprojections with the user sum. It does notMATERIALIZEthem, as 012 did not; older parts fall back to raw reads. - The unmerged #27877 also adds a 014. Migrations are tracked by file name, so both would run, but whichever merges second should become 015.
- 014 also rebuilds the
- Do not set
AI_USER_CREDITS_LIMITor a plan value during the rolling deploy of the first release that contains this. - Never revert, or deploy an older release, once rows with user credits exist in an open billing period: workspace allowance counters would rewarm at total spend.
- During the rolling deploy, an older front:
- reads the
workspaceAllowanceexhausted kind as unknown and shows the limit message instead of the upgrade banner; - still queries the old credit field names, so the billing page and the admin panel workspace billing tab fail to load until the browser reloads.
- reads the
- Any chat token or chat web search charge naming a user spends that user's credits. That includes impersonated chats, the creator's setup chat and app-sent chat turns.
- Trials get their plan's amount.
Not in this PR
- The front: user credits meter, banners and fallback setting. It must not rely on
SendChatMessageResult.hasWorkspaceCreditsLeftorAiChatUsage.userCredits, which were dropped. - Recording errored and interrupted chat turns. On main these turns keep their Redis debits without a usage row. The fix ships as its own PR off main.
USER_CREDITusage limit rows for admins.- Until the front lands, a
userAllowancerefusal shows the generic limit error, whose "Manage limits" button cannot change user credits, and the usage page shows 0 for a user who only spent user credits.
History
The feature went through about 27 commits of design changes, listed in earlier versions of this description. It was then simplified in #28045, merged here, and the branch is now squashed into one commit. The user allowance moved onto main's quota path instead of a parallel engine:
- one script instead of a second user-first script;
- one rule for who pays and what blocks;
- one
UsageLimitQuotaServiceinstead of five services; - no admission at job start;
- fewer AI chat fields.
| Before #28045 | Now | |
|---|---|---|
| Diff vs main, generated files excluded | 177 files, +7,955 / -3,488 | 133 files, +2,870 / -609 |
| Net usage-limit source | +1,511 | +458 |
| Net AI source | +502 | +109 |
Testing
-
Server typecheck is clean, and lint and format are clean on the changed files.
-
Unit specs pass in usage-limit, usage, billing, billing-webhook, admin-panel, twenty-config, logic-function, metadata-modules/ai, emailing and workflow-executor. Front AI util tests pass.
-
The extended Lua script ran against a local Redis on 12 scenarios (35 checks): plain limits, user pays all, exact split at the crossing, full window, STOP overshoot, two windows capped by the tightest, cold user key deferred then applied once on retry, cold workspace key, no workspace key, stock release and consume unchanged.
-
Integration cases:
- quota-consumption: user pays first, exact split at the crossing, STOP, and a failed warm;
- billing suite: the fallback setting (default, BILLING permission, refusal for a user without it).
The user credits cases need billing enabled, which CI integration runs do not have.
-
The failed-warm recovery ran against a local Redis with the real script and the real
consumeCounters, ClickHouse reads rejected: STOP with a cold user window, fallback with a cold workspace allowance, two user windows, no split, Redis down, and counters that cannot be built. The code before each fix fails the split cases and the Redis one. The updated integration case was not run locally (ClickHouse down, not enough free memory). -
No test guards that agent-run web searches are billed as
AI_WORKFLOW_WEB_SEARCH: the agent executor has no spec, and a user credits integration case needs billing enabled. -
QA on a local stack: real AI chat turns, a Stripe test-mode PRO trial synced by webhooks, ClickHouse 25.8 upgraded from main's schema with 014 applied on existing rows. Each case was checked against Redis, ClickHouse and the API.
- Upgrade: the instance command adds the column with
WORKSPACE_CREDITSon existing customers; usage recorded before 014 counts as workspace credits. - Unset limit: same as main, no user counter.
- User pays: whole turn and title on the user window (keyed to Monday 00:00 UTC), workspace allowance untouched.
- Crossing in spill mode: exact split (62 user + 56 workspace), one notice, then full spill with no second notice.
- Fallback setting: refused without BILLING, STOP survives a
customer.subscription.updatedwebhook, a flip reaches the worker immediately. - STOP: an exhausted user is refused at send with
userAllowance; a crossing tool-call step overshoots, the stream stops after it and the turn fails withUSER_CREDITS_EXHAUSTED. - Workspace credits exhausted: a user with credits keeps chatting and pays; generate-text is refused (402
workspaceAllowance). - User crossing while workspace credits are exhausted: split, notice with
isFallingBackToWorkspaceCredits: false,credits-exhausted, next send refused. - Workspace AI credit limit: warms to workspace credits only, is not debited or blocking while the user pays, still refuses generate-text (429).
- Chat web search: on user credits, exact split when it crosses the window.
- Plan values: a plan value applies with the instance variable unset; 0 disables user credits, STOP included.
- Setup chat with workspace credits exhausted: started and paid from the creator's user credits; unavailable when user credits are off.
- Counters equal the ClickHouse sums and rebuild to the same values after a drop.
- Billing page, usage page and admin panel workspace billing load and agree with the counters.
- Not covered: per-user CREDIT limits, which need the Organization plan.
- Upgrade: the instance command adds the column with