[BREAKING_CHANGE] feat(usage): spend user credits on AI chat and web search before workspace credits (server)

Merged
ET
1mo ago
+3382-875

Each user spends their own user credits on AI chat and its web searches before workspace credits are used. Nothing changes until AI_USER_CREDITS_LIMIT or a plan value is set. The front (meter, banners, fallback setting) comes in a follow-up PR.

How it works

Definition. User credits are a usage-limit quota default counted in the new UsageUnit.USER_CREDIT. AI declares one in USAGE_LIMIT_DEFINITIONS:

  • operations [AI_CHAT_TOKEN, WEB_SEARCH], sharing one counter (WEB_SEARCH is now AI chat search only);
  • one counter per user (spenderType: 'userWorkspace', isPerSpender: true);
  • a UTC calendar week;
  • the value from AI_USER_CREDITS_LIMIT in microCredits, which USAGE_LIMIT_DEFAULTS_BY_PLAN can set per plan. Unset or 0 means no user credits.

The engine is not AI specific: another resource gets user credits by declaring its own USER_CREDIT default. Several USER_CREDIT windows can apply at once, for example a day and a week. The user pays while every window has room.

Who pays. A charge that names a user and matches a USER_CREDIT quota on resource, operation and the userWorkspace spender is paid from user credits first.

  • User credits pay for chat turns and titles, and chat web search.
  • Workspace credits pay for:
    • agent runs, workflow AI agent steps, generate-text and AI evaluation (AI_WORKFLOW_TOKEN);
    • the web searches of agent runs and workflow AI agent steps (AI_WORKFLOW_WEB_SEARCH), even when a user started them;
    • agent triggers, which name no user;
    • app charges, logic functions and email, which have no USER_CREDIT default.

One quota path. The user allowance is a third quota counter kind, next to limits and the workspace allowance. It is built by the same builder, warmed from the same ClickHouse consumption rows, debited by the same script and admitted through the same path. UsageLimitQuotaService owns all of it, as on main. Reading the quota list now warms its counters in Redis, as a debit does, instead of only reading them.

Debit. One WARM_AND_CONSUME_COUNTERS_SCRIPT call per charge.

  • Keys are tagged user (each user window, with its limit) or workspaceCredits (the workspace allowance and workspace-wide CREDIT limits).
  • The script computes the user share as the smallest room left across the windows, capped at the amount. It debits every window by that share and the workspaceCredits keys by the rest, atomically.
  • If a tagged key is cold, no tagged key is debited until the retry has warmed them, so a share is never applied twice.
  • If the warm fails, the charge is admitted as on main and recorded as user credits, in both fallback modes. User windows still in Redis take the full amount, workspace keys take none, and a cold counter catches up from the recorded row when it warms. The same applies when the whole debit fails, for example Redis down: the charge is recorded as user credits. Only a failure before the counters are known, when it is not even known whether user credits apply, records it as workspace credits as on main. Workspace credits are otherwise never spent because a user window could not be read. In fallback mode, a user already out of room goes over their window for that charge instead of spilling.

What blocks. findBlockingCounters is shared by debit and admission.

  • While the user pays, the workspace allowance and workspace CREDIT limits do not block.
  • A full user window blocks only in STOP mode.

When user credits run out. The billing-owned setting billingCustomer.userCreditsFallback decides:

  • WORKSPACE_CREDITS (the default): charges spill to workspace credits.
  • STOP: the user's AI usage stops until the window renews (Monday 00:00 UTC for the AI default).

It is set with updateUserCreditsFallback (user sessions only, BILLING permission) and read from the cached currentBillingSubscription. The column comes with a 2.49 instance command.

Recording. usageEvent.userCreditsUsedMicro stores the user share, and creditsUsedMicro stays the total, so workspace credits are total minus user credits. computeRowConsumed holds the one rule:

  • USER_CREDIT counters count userCreditsUsedMicro;
  • workspace CREDIT limits count creditsUsedMicro - userCreditsUsedMicro.

The limit-form preview uses the same rule. Billing-period workspace and user credits come from one query.

AI chat.

  • The step debit returns { stopReason, userCreditsUsedMicro, userAllowanceExhaustedUntil }.
  • STOP ends the turn after the step that crosses the window.
  • The chat publishes user-credits-exhausted (isFallingBackToWorkspaceCredits, renewsAt) from the debit that crosses a window, either a step or the end-of-turn web search. isFallingBackToWorkspaceCredits is true only when nothing stops the turn: it is false in STOP mode and when the workspace allowance is exhausted by the same debit.
  • The turn records the summed user share.
  • New sends in STOP mode are refused by the send preflight. A queued turn that starts afterwards runs one step, then stops, as main does for the workspace allowance.
  • The workspace setup chat is admitted like a chat send of its creator (AI chat tokens, the creator as spender) instead of checking workspace credits only. A creator with user credits left can start it after workspace credits run out, and an exhausted AI usage limit now refuses it too.

API changes

  • BillingCustomer.userCreditsFallback, the UserCreditsFallback enum and the updateUserCreditsFallback mutation.
  • On BillingResourceCreditUsage and AdminPanelWorkspaceUsage, the workspace-only amounts are renamed: usedCredits, grantedCredits, rolloverCredits, totalGrantedCredits and remainingCredits (admin panel only) become workspaceCreditsUsed, workspaceCreditsGranted, workspaceCreditsExtraGranted, workspaceCreditsTotalGranted and workspaceCreditsRemaining. rolloverCredits was the sum of every active credit grant, not only rollovers, hence workspaceCreditsExtraGranted. userCreditsUsed is added.
  • UsageBreakdownItem gets userCreditsUsed and workspaceCreditsUsed, and UsageTimeSeries gets workspaceCreditsUsed. creditsUsed stays the total on both.
  • Every credits query in UsageAnalyticsService computes workspace credits in SQL and still returns the total. It subtracts the summed aliases (creditsUsedMicro - userCreditsUsedMicro): ClickHouse binds a column name to a same-named alias, so sum(creditsUsedMicro) - sum(userCreditsUsedMicro) next to sum(creditsUsedMicro) AS creditsUsedMicro fails as a nested aggregate. The usage page shows workspace credits only, and breakdowns are ordered by them. The admin AI usage by workspace keeps showing the total, since it tracks AI cost.
  • UsageOperationType.AI_WORKFLOW_WEB_SEARCH for the web searches of agent runs, so automation searches stay on workspace credits, like their tokens. They keep the user who started them, so per-member limits and usage still count them. Admins can limit it like WEB_SEARCH. An existing WEB_SEARCH limit no longer counts agent-run searches; limits on all AI operations still do. Prod has two WEB_SEARCH limits, both workspace-wide, on workspaces with no agent runs and no web search in the last 60 days.
  • UsageUnit.USER_CREDIT. It is not in any allowedUnits, so admins cannot create USER_CREDIT usage limit rows yet.
  • The exhausted kind allowance becomes workspaceAllowance, and userAllowance is added. Front checks are updated. The workspace allowance Redis key string is unchanged, so live counters stay warm.
  • New subscription event user-credits-exhausted.
  • The removed BillingResourceCreditUsage fields fail the breaking-changes check; the [BREAKING_CHANGE] title tag accepts them.

Vocabulary

  • Credits are workspace credits or user credits. Their counters are the workspace allowance and the user allowance.
  • Plain "credits" only names a value holding both.

Rollout constraints

  • Migration 014 must run before this code deploys. Every usage insert now writes userCreditsUsedMicro, and the readers use it. ClickHouse migrations run after code deploys, so 014 has to land first on its own, or be run by hand.
    • 014 also rebuilds the consumption_by_scope and billing_by_workspace_period projections with the user sum. It does not MATERIALIZE them, as 012 did not; older parts fall back to raw reads.
    • The unmerged #27877 also adds a 014. Migrations are tracked by file name, so both would run, but whichever merges second should become 015.
  • Do not set AI_USER_CREDITS_LIMIT or a plan value during the rolling deploy of the first release that contains this.
  • Never revert, or deploy an older release, once rows with user credits exist in an open billing period: workspace allowance counters would rewarm at total spend.
  • During the rolling deploy, an older front:
    • reads the workspaceAllowance exhausted kind as unknown and shows the limit message instead of the upgrade banner;
    • still queries the old credit field names, so the billing page and the admin panel workspace billing tab fail to load until the browser reloads.
  • Any chat token or chat web search charge naming a user spends that user's credits. That includes impersonated chats, the creator's setup chat and app-sent chat turns.
  • Trials get their plan's amount.

Not in this PR

  • The front: user credits meter, banners and fallback setting. It must not rely on SendChatMessageResult.hasWorkspaceCreditsLeft or AiChatUsage.userCredits, which were dropped.
  • Recording errored and interrupted chat turns. On main these turns keep their Redis debits without a usage row. The fix ships as its own PR off main.
  • USER_CREDIT usage limit rows for admins.
  • Until the front lands, a userAllowance refusal shows the generic limit error, whose "Manage limits" button cannot change user credits, and the usage page shows 0 for a user who only spent user credits.

History

The feature went through about 27 commits of design changes, listed in earlier versions of this description. It was then simplified in #28045, merged here, and the branch is now squashed into one commit. The user allowance moved onto main's quota path instead of a parallel engine:

  • one script instead of a second user-first script;
  • one rule for who pays and what blocks;
  • one UsageLimitQuotaService instead of five services;
  • no admission at job start;
  • fewer AI chat fields.
Before #28045Now
Diff vs main, generated files excluded177 files, +7,955 / -3,488133 files, +2,870 / -609
Net usage-limit source+1,511+458
Net AI source+502+109

Testing

  • Server typecheck is clean, and lint and format are clean on the changed files.

  • Unit specs pass in usage-limit, usage, billing, billing-webhook, admin-panel, twenty-config, logic-function, metadata-modules/ai, emailing and workflow-executor. Front AI util tests pass.

  • The extended Lua script ran against a local Redis on 12 scenarios (35 checks): plain limits, user pays all, exact split at the crossing, full window, STOP overshoot, two windows capped by the tightest, cold user key deferred then applied once on retry, cold workspace key, no workspace key, stock release and consume unchanged.

  • Integration cases:

    • quota-consumption: user pays first, exact split at the crossing, STOP, and a failed warm;
    • billing suite: the fallback setting (default, BILLING permission, refusal for a user without it).

    The user credits cases need billing enabled, which CI integration runs do not have.

  • The failed-warm recovery ran against a local Redis with the real script and the real consumeCounters, ClickHouse reads rejected: STOP with a cold user window, fallback with a cold workspace allowance, two user windows, no split, Redis down, and counters that cannot be built. The code before each fix fails the split cases and the Redis one. The updated integration case was not run locally (ClickHouse down, not enough free memory).

  • No test guards that agent-run web searches are billed as AI_WORKFLOW_WEB_SEARCH: the agent executor has no spec, and a user credits integration case needs billing enabled.

  • QA on a local stack: real AI chat turns, a Stripe test-mode PRO trial synced by webhooks, ClickHouse 25.8 upgraded from main's schema with 014 applied on existing rows. Each case was checked against Redis, ClickHouse and the API.

    • Upgrade: the instance command adds the column with WORKSPACE_CREDITS on existing customers; usage recorded before 014 counts as workspace credits.
    • Unset limit: same as main, no user counter.
    • User pays: whole turn and title on the user window (keyed to Monday 00:00 UTC), workspace allowance untouched.
    • Crossing in spill mode: exact split (62 user + 56 workspace), one notice, then full spill with no second notice.
    • Fallback setting: refused without BILLING, STOP survives a customer.subscription.updated webhook, a flip reaches the worker immediately.
    • STOP: an exhausted user is refused at send with userAllowance; a crossing tool-call step overshoots, the stream stops after it and the turn fails with USER_CREDITS_EXHAUSTED.
    • Workspace credits exhausted: a user with credits keeps chatting and pays; generate-text is refused (402 workspaceAllowance).
    • User crossing while workspace credits are exhausted: split, notice with isFallingBackToWorkspaceCredits: false, credits-exhausted, next send refused.
    • Workspace AI credit limit: warms to workspace credits only, is not debited or blocking while the user pays, still refuses generate-text (429).
    • Chat web search: on user credits, exact split when it crosses the window.
    • Plan values: a plan value applies with the instance variable unset; 0 disables user credits, STOP included.
    • Setup chat with workspace credits exhausted: started and paid from the creator's user credits; unavailable when user credits are off.
    • Counters equal the ClickHouse sums and rebuild to the same values after a drop.
    • Billing page, usage page and admin panel workspace billing load and agree with the counters.
    • Not covered: per-user CREDIT limits, which need the Organization plan.