feat(app): PostHog paywall A/B test (control / outline / web) + flag-gated web onboarding

Merged
KO
3d ago
+3018-110

What changed and why

Mobile free→paid conversion is ~20% against ~50% for Pocket/Plaud. Nik is trialing several paywall redesigns and wants each one split-tested through PostHog, switchable per variant, and ideally changeable over the air. This PR adds that framework and the first variant.

Paywall A/B test (paywall-variant, PostHog multivariate flag 957521, experiment 482365, dashboard 2207880)

  • PaywallExperiment resolves the arm once per session from the flag + payload; anything unexpected is control. Debug builds can force an arm with --dart-define=OMI_PAYWALL_VARIANT=outline|web.
  • PaywallRouter.openVariant sits in front of every opener (usage page incl. the locked-row/deep-link routes, chat quota, freemium meter, contextual plans). Control, paid readers and hidden-subscription-UI builds (App Review) return false and the PlansSheet opens exactly as before.
  • outline = native port of the soft paywall from #21412 (Graphite direction) on the real catalog (Unlimited tier, yearly/monthly, saving derived from prices) and the shared checkout (PaywallCheckout, the same Stripe session + webview + analytics the sheet uses). "Use Free Plan" is always visible.
  • web = a webview rendering the page named in the payload (web/paywall/index.html, deployed at https://omi-paywall.vercel.app). The page only renders; checkout stays native via a JS bridge. Dead page (error, 4xx/5xx, 10 s silence) falls back to the native sheet (forceNativePaywall).
  • Payload onboarding_paywall: true adds a placement at the end of first-run onboarding (variants only; control is unchanged).
  • Exposures: $feature_flag_called once per analytics identity and arm, control included (the denominator), recorded by the router right before any paywall opens; the SDK's own exposure events are off. PostHog Experiments attribute Upgrade Succeeded, new MRR, dismissals and Product Error per arm; $feature/paywall-variant is registered as a super property for breakdowns. New events Paywall Variant Shown / Paywall Variant Fallback; onboarding added to the paywall source enums.

Web onboarding (onboarding-webview, flag 957522, 0%)

  • After sign-in and consent, OnboardingWebviewGate can route to WebOnboardingPage (web/paywall/onboarding.html). Bridge: set_name, permission (notifications/location/background via the existing PlatformOnboardingPermissionsSource), complete, fallback. Completion stays in the wrapper (_finishOnboarding, shared with the native completion screen). Any failure lands on the native step the reader left off at.
  • Debug builds: --dart-define=OMI_ONBOARDING_WEBVIEW_URL=https://… and OMI_ONBOARDING_RERUN=1 to walk onboarding again on a signed-in account.

Adding a variant later: native = one new widget + a PaywallVariant case + a variant row in the flag; web = edit the page and redeploy, no app release.

Both flags are live with control at 100% / rollout 0%, so no user sees any change until the split is set once this build is in the stores.

Product invariants affected

none

How it was verified

Simulator (iPhone 16 Pro, dev flavor built against the production profile + api.omi.me with a local, uncommitted profile-guard bypass; signed in as the free test account nikbasedhardware@gmail.com via a Firebase custom token through the VM service, the repo's documented simulator workaround). Screenshots in .agent-artifacts/ui-evidence/paywall-ab/ (evidence.json inside).

Outline paywall at the end of onboarding
Web paywall from Plan & Usage
Web onboarding

  • --dart-define=OMI_ONBOARDING_RERUN=1 OMI_ONBOARDING_WEBVIEW_URL=https://omi-paywall.vercel.app/onboarding OMI_PAYWALL_VARIANT=outline: after launch the web onboarding page rendered inside the app with the name pre-filled; Continue → step 2 greeted by name (set_name reached native); Location → native request → status posted back and the page re-labelled the button; Start using Omi → native completion ran and the end-of-onboarding placement showed the Outline paywall with the prod catalog (Unlimited $269.91/yr shown as $22.49/mo, $29.99/mo, Save ~25%, 300 free minutes from the subscription). Upgrade to Unlimited → real Stripe Checkout for Omi Unlimited v2 at $269.91/yr in the shared payment webview (cancelled, nothing paid) → back on the paywall → Use Free Plan → Home.
  • Same build, in-app entry: Settings → Plan & Usage → Upgrade → Outline paywall; X → back to Plan & Usage.
  • OMI_PAYWALL_VARIANT=web: Plan & Usage → Upgrade → the webview paywall (omi-paywall.vercel.app) with prices and name from the bridge URL and the native close overlay; its Upgrade button → the same native Stripe session (cancelled) → Use Free Plan → Plan & Usage.
  • Analytics (same build with the PostHog project token in the local dev env): after opening and dismissing the paywall from Plan & Usage, PostHog project 302298 received for the test account, within ~1 minute: $feature_flag_called ($feature_flag=paywall-variant, $feature_flag_response=outline), Paywall Opened (source usage_page), Paywall Variant Shown (variant outline, source usage_page) and Upgrade Modal Dismissed, every one carrying $feature/paywall-variant. (That property read control in the debug run because the SDK reports the server's assignment while the dart-define forced the arm; without the override both agree.)
  • flutter analyze clean on every touched file; flutter test: 40 unit/widget tests pass (paywall_experiment_test 13 incl. identity scoping and mid-read account switch, paywall_router_test 3 incl. control exposure and hidden/paid readers, analytics_is_feature_enabled_test incl. per-identity exposure dedupe, onboarding_webview_gate_test 5, outline_paywall_page_test 4, onboarding_setup_rating_prompt_gate_test). make preflight passes (ui-evidence included).
  • PostHog objects created with control at 100%: flag 957521 (paywall-variant, payloads set), flag 957522 (onboarding-webview, 0%), experiment 482365 (draft, not started), dashboard 2207880 (every metric broken down by $feature/paywall-variant).

Not verified: a completed purchase (no live charge was made); the chat-quota and freemium-meter openers only through the shared router (same code path as the usage page); Android.

Failure class

none

Exposure dedupe, verified on device (Oct 14, prod backend, uid nikbasedhardware): the first $feature_flag_called per identity and arm reaches PostHog and spawns $experiment_exposure (fresh arms sent through recordFeatureFlagExposure today all landed within seconds). A repeat of an already-reported (identity, flag, response) pair is dropped downstream of the app (SDK/PostHog side — a raw HTTP repeat lands, the same event through the SDK does not), so the in-app once-per-identity set and PostHog agree: exposure counts once per person per arm, which is exactly what the experiment needs.