feat(desktop): three-arm new-workspace prompt form-factor experiment

Merged
SA
3mo ago
+266-121

Reworks the new-workspace sample prompts so the prompt-cards experiment measures form factor and nothing else, and adds a third arm to test whether four cards cost more than they earn.

Why this is safe to change

The experiment (PostHog 419092) never launched — still draft, flag active: false. All 639 $feature_flag_called evaluations since desktop-v1.22.0 returned null, so nobody has ever been bucketed into an arm. Zero experiment data exists, and #6537 / #6540 are in no shipped desktop tag.

What had drifted

Pre-registration said control = today's three inline rows, unchanged; test = two cards. On main:

Pre-registeredOn main before this PR
control3 fixed rows4 shuffled from a pool of 6
test2 bordered cards4 borderless cards, shuffled
selectionidentical across armssetup-first in cards only

That last row is the blocker: the cards arm got a curated setup-first list and rows got a random slice, so the arms differed in content inside a layout test.

Premises, validated against production

  • Setup is 69.5% of clicks, not 75% — but it always renders first, so share and position are perfectly confounded.
  • Setup clickers create ~3x more workspaces (medians), and only 12.7% of those predate the click. But the cold-start cell is n=17, and fix-small-bug — the least-clicked prompt — shows the highest count, which is a selection signature. Suggestive, not established.
  • Eligible population is ~840 new accounts/week at a stable 13.5–14.1% click rate. Activation can't resolve a realistic form-factor effect in under ~10 weeks; click rate resolves a large relative effect in ~2–3 weeks across three arms.

So click rate becomes the primary metric and activation is demoted to directional. Powering this on activation is what made a 3-arm test look unaffordable.

Changes

  • Fixed five-prompt pool, curated order. Arms slice nested prefixes (cards2 ⊂ control ⊂ cards4), so only form factor varies. clean-up-todos cut — with fixed order and nesting, only five prompts are ever reachable.
  • Shared useSamplePromptSelection — setup-first rule and its pending gate now apply identically to every arm. Previously cards held for the setup query while rows rendered instantly, a timing difference the experiment would have read as a layout effect.
  • Three arms: control (3 rows) / cards2 / cards4. layout on the click event distinguishes them.
  • Card border restored — prominence is the manipulation under test; #6540 had slid it to borderless across three commits.
  • Shuffle dropped — randomizes content against a tight power budget and breaks the nesting.
  • Dismiss gated behind the user's first non-main workspace. It permanently removes the measured surface, and asymmetric dismissal would bias the primary metric toward null. (main is auto-created for every account, hence the != main condition.)

Kept from #6537/#6540: the expanded pool (a 4-card arm can't be filled from three prompts), per-card icons, AgentLogoCluster, and the labeled model/effort pills.

Before this can launch

  1. Rename the flag's variants control / cards2 / cards4 at 33/33/33 and re-register the experiment with click rate primary. Not done in this PR — no unapproved tracker writes.
  2. Bump the flag's created_at cutoff to the carrying build's publish time (currently 2026-08-17T07:00Z, already in the past).

@Kitenite — this reworks parts of #6537 and #6540, so please take a look. The reasoning is measurement-driven rather than taste-driven; your cards and icons are kept.

bun run typecheck, ./scripts/lint.sh, and 2583 desktop tests pass.


Summary by cubic

Aligns the new-workspace prompt experiment to measure form factor only, adds a third arm, and introduces tiered prompts keyed on whether the user has shipped a workspace. Old behavior randomized content and timing with one cold-start pool; new behavior fixes order, shares selection and timing across arms, restores card borders, distinguishes cards2 vs cards4, and switches between first-run and returning tiers to avoid repeating cold-start prompts.

  • Arms: control shows 3 rows; cards2 shows 2 cards; cards4 shows 4 cards. All arms slice nested prefixes of one ordered tier list; the setup prompt drops if the project is configured.
  • Shared useSamplePromptSelection applies the same setup-first rule and pending gate to every arm, so rows no longer appear earlier than cards.
  • Tiering: first-run keeps orientation; returning adds explain-superset and restores clean-up-todos. Click events carry layout (rows/cards2/cards4) and tier.
  • Shuffle is dropped; cards regain borders; dismiss events use the same layout. The dismiss button appears only after the user has a non-main workspace.
  • Flag reading accepts cards2/cards4 variants; the manual override maps to cards2.

Rollout

  • Rename the flag’s variants to control, cards2, cards4 at 33/33/33 and re-register the experiment with click rate as the primary metric.
  • Bump the flag’s created_at cutoff to the carrying build’s publish time.

Summary by CodeRabbit

  • New Features

    • Added updated workspace suggestion layouts, including two- and four-card formats.
    • Suggestions now adapt to project setup and whether you’re creating a workspace for the first time or returning.
    • Added an accessible dismiss button for hiding suggestions.
  • Bug Fixes

    • Improved prompt loading behavior to prevent incomplete suggestions from appearing.
    • Dismissal is available only when appropriate and accurately tracks the displayed layout.
    • Updated prompt cards with refreshed styling and clearer actions.