feat: public agent usage leaderboard

Merged
HA
2mo ago
+14796-102

What & why

An opt-in public leaderboard for agent usage, so engineers can see how much
agent work they're running relative to everyone else.

What gets published. Token counts, model names, session counts and a USD
estimate — never repo names, file paths or prompts. Opt-in is explicit, the
handle is chosen by the user, and leaving deletes every published row rather
than hiding it (FK cascade from leaderboard_participants). The join dialog
states that the account name is shown publicly alongside the handle, because
the public API serves it.

The pieces:

  • packages/db — migration 0094 adds leaderboard_participants,
    leaderboard_daily and leaderboard_daily_factory, plus a
    leaderboard_visibility enum. Generated with drizzle-kit, not hand-edited.
  • packages/trpc — leaderboard router: join / leave / setVisibility /
    publish / standings / previewRank / me, rate-limited per user through
    Upstash. Response shapes live in a db-free types.ts so consumers can import
    them as types without pulling in the db client.
  • packages/host-service — builds the publish payload from local transcripts.
    The scan-and-parse half of computeUsageHistory is extracted into
    entries.ts so the leaderboard payload and usage history share one
    implementation instead of two copies of the profile-discovery logic.
  • apps/api — public, cached, CORS-open GET /api/leaderboard,
    /api/leaderboard/stats and /api/leaderboard/u/[handle].
  • apps/desktop — opt-in from Settings → Account or the Usage tab, a first-run
    prompt, and a 2h auto-publish loop that skips the round-trip when the payload
    hash is unchanged.
  • apps/marketing — /leaderboard, /stats and /user/[handle] (with OG
    images), all server-rendered on a 5 min revalidate.

Factory tiers. Participants get a 1–4 tier over a rolling 30-day window,
gated by the weakest of four axes (width, depth, output, sustain) so that
improving an axis already ahead of the others moves nothing. Tier is
deliberately not re-sliced by the selected period — it is already a 30-day
figure, and re-windowing it would report a number that means nothing.

How I tested it

  • bun run lint — clean across 6533 files (CI fails on warnings, so this is
    the gating check).
  • bun run typecheck — passes for every package. The one failure on main
    (@superset/api can't resolve resend) predates this branch and is
    untouched by it.
  • bun test — 73 new tests across 6 files covering tier computation and
    progress, period/day-range resolution, day bucketing, factory-day rollups,
    the auto-publish schedule and the first-run flag. The persisted-key registry
    test passes, so the two new localStorage keys are properly allowlisted.
  • bunx sherif — 0 errors.

Not yet done: no CDP screenshots of the desktop opt-in flow or the three
marketing pages. Everything user-visible here is unverified in a running app —
worth a look before this merges.

Review fixes

The first review pass turned up three issues, all fixed in 61834e0d0:

  • Deleted accounts leaked into public aggregates. getStats didn't filter
    users.deletedAt the way getStandings and getParticipant do, so a
    deleted account's tokens still counted toward the public totals and the tier
    distribution. The windowed participant count inside getStandings had the
    same gap, which additionally made total/hasMore disagree with the rows
    returned beside them.
  • Unbounded scan from the public API. resolveWindow now clamps an
    explicit from/to to MAX_WINDOW_DAYS (366), so an unauthenticated caller
    can no longer ask for a full-table scan of leaderboard_daily. Clamping
    rather than rejecting matches how limit/offset are already handled on
    these routes.
  • Handle race returned the wrong error. A handle claimed between join's
    pre-check and its insert raised a raw Postgres unique violation; it now
    surfaces as CONFLICT, same as the pre-check path. The detection walks the
    error's cause chain because drizzle wraps driver errors.

Checklist

  • PR title follows conventional commits (type(scope): subject)
  • bun run lint and bun run typecheck pass (CI fails on lint warnings too)
  • "Allow edits from maintainers" is checked on fork PRs

Summary by CodeRabbit

  • New Features

    • Added public leaderboard pages with rankings, statistics, filters, tier progress, token breakdowns, and participant profiles.
    • Added profile sharing, social links, avatars, and social preview images.
    • Added desktop leaderboard opt-in, privacy controls, suggested handles, rank previews, first-run prompts, and automatic usage publishing.
    • Added tier badges, progress visuals, responsive statistics, and improved usage formatting.

  • Bug Fixes

    • Improved token formatting around unit-rounding boundaries.
    • Added graceful handling for unavailable statistics and missing profiles.

  • Tests

    • Added coverage for leaderboard aggregation, scheduling, tiers, date ranges, persistence, and formatting.