Playground build-out: state, events, data, components and rooms (P2-2)

Merged
FA
12d ago
+4188-42

All Submissions:

  • Have you followed the guidelines stated in CONTRIBUTING.md?
  • Have you checked to ensure there aren't any other open Pull Requests for the desired changed?

Type of change

  • New feature (non-breaking change which adds functionality): the benchmark fixture app grows.
  • Bug fix (non-breaking change which fixes an issue): AppHarness no longer leaks the states of an app's submodules into the next app (see "Follow-up commits").
  • This change requires a documentation update (done: both READMEs).

New Feature Submission:

  • Does your submission pass the tests?
  • Have you linted your code locally prior to submission?

Phase-2 task P2-2 of Holistic Framework Benchmarks: ENG-12606 (state, events, data), ENG-12607 (components and compiler surface) and ENG-12609 (collaboration), in one PR so the playground's content hash, and with it every playground benchmark series, resets once. After this PR the playground stays fixed.

What changed and why

examples/playground: 4 routes become 21.

  • State and events (ENG-12606). There are 13 new substates under rx.State in playground/states/, with:
    • field types: scalars, lists, dicts, a dataclass (Task), an rx.Model field (ProductEditState.product), rx.Field/rx.field, and backend-only _ vars;
    • mutations through the mutable proxy: append, dict[key] = ..., and in-place edits of dataclass rows and nested lists;
    • computed vars: cheap ones, an expensive one (a sort over all 2,000 rows), and a chain of three (TasksState.done_count → done_share → progress_label);
    • handlers: sync, async, generator, background, chained, with arguments, server-side (rx.toast, rx.redirect, rx.call_script, rx.download), and get_state of a sibling substate (EventsState.read_tasks);
    • on_load handlers; /data's handler runs the first query.
  • Data (ENG-12606).
    • Product(rx.Model) over sqlite, with db_url set in rxconfig.py.
    • playground/seed.py builds 2,000 rows from random.Random(20260929) using only the stdlib. A sync lifespan task (seed_database) creates the table and inserts the rows with one executemany in one transaction, only into an empty table (INSERT OR IGNORE, so two workers can't double-insert).
    • The data pages add a table with filter, sort and pagination; a detail page with create, update and delete; validated forms; rx.upload with progress; and rx.Cookie, rx.LocalStorage and rx.SessionStorage on /storage.
  • Components and compiler surface (ENG-12607).
    • Pages: 21 routes, a nested layout (data_layout inside layout()), a custom 404, and two new dynamic routes (/data/product/[product_id], /room/[room_id]).
    • Rendering: rx.foreach (nested included), rx.cond, rx.match, rx.memo, rx.ComponentState, rx.dynamic, and a custom rx.Component using add_imports, add_custom_code and add_hooks.
    • Packages: rx.recharts, rx.plotly, rx.data_table, rx.data_editor, rx.markdown, rx.code_block, rx.icon, rx.moment, rx.audio/rx.video, rx.toast, and radix layout and controls.
    • App level: an appearance toggle (on /settings), a raw ASGI api_transformer adding GET /api/playground/stats, the seeding lifespan task, and theming.
    • Tailwind: PLAYGROUND_TAILWIND (default 1) adds TailwindV4Plugin(). /about and the data sidebar use Tailwind classes.
  • Collaboration (ENG-12609).
    • /room/[room_id] joins on load (RoomState.enter → BoardState.enter_room, which calls _link_to) and shows presence (BoardState.members), a shared counter, +10 at once (contention: every visitor mutates one token) and Broadcast (fan-out through set_seq_shared).
    • /board keeps #board-join (join("lobby")) and adds #board-token + #board-join-token.
    • The README documents it as a demo.
  • Benchmark contract unchanged. PlaygroundState, BenchState and BoardState stay in playground/state.py as direct children of rx.State, and nothing was added to BenchState. BoardState gains members plus enter_room/leave_room; join and set_seq_shared are unchanged. Element ids, routes, the three pragma lines, .bench-hooks and mark.svg are unchanged, and marker() is still imported only by the index page. / keeps its content plus the longer navigation bar.
  • README. It now lists the pages, the room demo, every page's element ids, the size counts, the db/plotly dependencies and the Tailwind switch. The benchmark contract section gains the names the harness reads but the old contract missed: #count/#increment on /counter (hmr.reconnect), the playground.state import (STATES_SCRIPT), .bench-hooks' literal 0.75rem (hmr.css), components/marker.py (lifecycle.compile.incremental), and the /counter and /item/42 routes (wire.navigate).

Harness (packages/reflex-bench).

  • SUBJECT_REQUIREMENTS = ("sqlmodel>=0.0.24,<0.1", "alembic>=1.15.2,<2.0", "plotly>=6.0,<7.0"). It is part of every venv cache key, so every cached subject venv is rebuilt once.
  • plotly was added per D1: on 0.8.23 rx.plotly fails at compile with ModuleNotFoundError: No module named 'plotly' (reflex/components/plotly/plotly.py:217, from plotly.graph_objs.layout import Template in create), and HEAD imports it in create too. It was also added to the playground's pyproject.toml (reflex[db]>=0.8.23, plotly>=6.0,<7.0) and to the 0.8.23 install line of examples.yml.
  • budgets.json is set to the measured values below.
  • The WireSession.navigate docstring and the README (Fixtures, Wire sizes, Subjects) are updated for D2 and the new fixture.
  • No benchmark class changed.

Tests.

  • tests/units/reflex_bench/test_subjects.py: the requirements, and that they reach the install command of PyPI, git and path subjects. These tests were written before the change.
  • tests/units/reflex_bench/fixtures/test_playground.py: loads seed.py by path, checks two calls return identical rows and a fixed digest (also verified on Python 3.12), and that the module imports only random.
  • tests/units/reflex_bench/fixtures/test_playground_app.py: an opt-in console sweep (REFLEX_BENCH_APP_TESTS=1, plus REFLEX_BENCH_NETWORK_TESTS=1 for 0.8.23). It loads the explicit route list in dev and prod, fails on any page or console error, and checks that the README's route table lists the same routes. It also checks /api/playground/stats reports the 2,000 seeded products. test_interactions_work (added after review) drives the pages' controls.
  • tests/units/reflex_bench/suites/test_wire_app.py: the hydration of / now receives 5 frames instead of 4 (see "rx.dynamic" below).

Verification

1. Compile on HEAD and on 0.8.23 (Python 3.12)

$ uv run --directory examples/playground --project ../.. reflex --version
0.9.12.post37.dev0+50cb0ce35
$ uv run --directory examples/playground --project ../.. reflex compile
DeprecationWarning: reflex.Model has been deprecated in version 0.9.2. Directly 
use database ORM layer, like sqlalchemy or SQLModel. It will be completely 
removed in 1.0.0. 
(/home/user/reflex/.venv/lib/python3.14/site-packages/pydantic/_internal/_model_
construction.py:156)
Warning: The _evaluate method is experimental and may be removed in future 
versions.
Success: App compiled successfully in 2.075 seconds.

$ reflex --version  # reflex==0.8.23 venv, Python 3.12
0.8.23
Python 3.12.3
$ reflex compile
Warning: The _evaluate method is experimental and may be removed in future 
versions.
[00:07:20] Compiling: ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 100% 97/97 0:00:01
Success: App compiled successfully in 3.299 seconds.

Two compiles of the same app give identical .web sources on both versions (0.8.23 differs only in its reflex.install_frontend_packages.cached marker).

2. Console sweep and interactions: HEAD and 0.8.23

Latest run, on a6acf6ee5:

$ REFLEX_BENCH_APP_TESTS=1 REFLEX_BENCH_NETWORK_TESTS=1 uv run pytest tests/units/reflex_bench/fixtures/test_playground_app.py tests/units/reflex_bench/fixtures/test_playground.py -v
tests/units/reflex_bench/fixtures/test_playground_app.py::test_the_readme_lists_every_route PASSED [ 11%]
tests/units/reflex_bench/fixtures/test_playground_app.py::test_every_route_loads_without_errors[workspace-dev] PASSED [ 22%]
tests/units/reflex_bench/fixtures/test_playground_app.py::test_every_route_loads_without_errors[workspace-prod] PASSED [ 33%]
tests/units/reflex_bench/fixtures/test_playground_app.py::test_every_route_loads_without_errors[0.8.23-dev] PASSED [ 44%]
tests/units/reflex_bench/fixtures/test_playground_app.py::test_every_route_loads_without_errors[0.8.23-prod] PASSED [ 55%]
tests/units/reflex_bench/fixtures/test_playground_app.py::test_interactions_work[workspace] PASSED [ 66%]
tests/units/reflex_bench/fixtures/test_playground_app.py::test_interactions_work[0.8.23] PASSED [ 77%]
tests/units/reflex_bench/fixtures/test_playground.py::test_seed_rows_are_deterministic PASSED [ 88%]
tests/units/reflex_bench/fixtures/test_playground.py::test_seed_module_imports_only_the_stdlib PASSED [100%]
======================== 9 passed in 396.27s (0:06:36) =========================

test_interactions_work[workspace] also passes with REFLEX_REDIS_URL=redis://localhost:6379 (30 keys written).

The sweep allows exactly one console message: Failed to load resource: ... status of 404, on the 404 route, and in prod on dynamic routes. Reflex's prod server answers every dynamic route with status 404 plus the app shell, and the playground on main does the same (curl against HEAD prod on main's playground: / 200, /counter 307, /item/42 404).

3. Collaboration: two browser contexts in one room, per state manager (prod)

reflex 0.9.12.post37.dev0+50cb0ce35
memory: ok (presence 2 -> 1, count 0 -> 1 -> 11, broadcast seen)
disk:   ok (presence 2 -> 1, count 0 -> 1 -> 11, broadcast seen)
redis:  ok (presence 2 -> 1, count 0 -> 1 -> 11, broadcast seen)
reflex 0.8.23
memory: ok (presence 2 -> 1, count 0 -> 1 -> 11, broadcast seen)
disk:   ok (presence 2 -> 1, count 0 -> 1 -> 11, broadcast seen)
redis (GRANIAN_WORKERS=1): ok (presence 2 -> 1, count 0 -> 1 -> 11, broadcast seen)

This check ran before the review fixes. The committed test_interactions_work now covers the same two-visitor flow (disk on HEAD and 0.8.23, redis on HEAD).

On 0.8.23 with redis and the default worker count, even the existing /board counter does not sync: after one client incremented, the other still showed 0. With several granian workers, 0.8.23 fans shared-state updates out only to sessions on the same worker. HEAD reaches every worker. The README says so, and the event benchmarks already run one worker.

4. Counts (targets vs actual)

TargetActual
Pages15–2521
Substates (new)8–1213 new (16 in total), plus 2 rx.ComponentState instances
Event handlers~6082 (70 new)
Computed vars~1520 (14 new)
Component instances2–4k2,986 (95 on /item/[item_id] to 236 on /data)

The count script walks each page function's children recursively and counts states, handlers and computed vars defined under playground. It ran with the workspace interpreter from examples/playground and is not committed.

5. Benchmarks

Daily suite, smoke, this branch. The machine is a 4-core Intel Xeon VM (reflex-bench doctor flags it as a VM). Its cgroup v2 hierarchy has no memory controller, so memory.boot_512mb cannot run here.

  • Workspace (HEAD): 56 ok, 2 failed. Neither failure is this PR's:
    • hmr.css fails the same way on main on this machine (DroppedUpdateError: vite sent the hot update, the page kept 12px).
    • memory.boot_512mb fails with cgroup scopes are unavailable ... (hybrid cgroup v1 host).
  • 0.8.23 / Python 3.12: 45 ok, 8 unsupported (all .preview, which needs 0.9.8), 5 failed:
    • hmr.render.root and both memory.leak entries fail the same way on main on this machine. On main, leak slopes are 199.8 B/event (memory) and 211.6 B/event (disk); on this branch, 186.0 and 189.8.
    • memory.boot_512mb: no cgroup memory controller, as above.
    • events.shared_contention.latency[rate=auto] failed on the generator's own lag check (p99 2.50 ms > 1 ms). Rerun 3× from this branch and 3× from main on 0.8.23: each side passed 2 of 3.

Fixed-rate latency. events.simple.latency[manager=disk,sessions=10,rate=500] leaves nothing unanswered:

  • HEAD: unanswered 0, 500 ev/s answered, p99 3.8 ms (capacity 1,156 ev/s in the same run);
  • 0.8.23: unanswered 0, 498.4 ev/s answered, p99 11.5 ms (capacity 716 ev/s).

Opt-in suite app tests. REFLEX_BENCH_APP_TESTS=1 REFLEX_BENCH_NETWORK_TESTS=1 uv run pytest tests/units/reflex_bench/suites -q (includes test_browser_hmr_app_0823.py): 244 passed, 2 failed, 1 skipped in 16 min 45 s. After fixing the frame count, both failures pass (4 passed):

  • test_wire_app.py::test_navigate_for_real expected 4 hydration frames; it is 5 now, and the test is updated.
  • test_events_app.py::test_simple_events_for_real[0.8.23] failed the generator's lag check (1.26 ms > 1 ms).

Before/after, same machine. Default policy, from an origin/main worktree and from this branch. The fixture hash differs, so the medians are shown side by side:

BenchmarkMetricmainthis branchchange
lifecycle.compile.warm[app=playground]wall0.866 s (n=30)2.581 s (n=12)+198 %
lifecycle.compile.warm[app=playground]peak_mem84,186,112 B (n=30)164,748,800 B (n=12)+96 %
lifecycle.run.prod.ready[app=playground]http_ready4.066 s (n=10)13.684 s (n=10)+237 %
size.export[app=playground]initial_gzip311,746 B (n=1)340,727 B (n=1)+9 %
size.export[app=playground]total_gzip339,513 B (n=1)4,199,360 B (n=1)+1137 %
size.export[app=playground]chunks17 (n=1)401 (n=1)+2259 %
size.export[app=playground]node_modules140,168,718 B (n=1)632,174,766 B (n=1)+351 %
wire.hydratehydrate_received_bytes2,571 B (n=1)9,954 B (n=1)+287 %
wire.hydratehydrate_frames4 (n=1)5 (n=1)+25 %
events.simple.capacity[manager=disk,sessions=10]throughput3,121 ev/s (n=10)1,269 ev/s (n=10)-59 %
  • compile.warm ~3×. Expected: 21 pages instead of 4. It now takes seconds.
  • run.prod.ready +9.6 s. Almost all of it is the bigger prod frontend build (plotly.js, the grids and the rest). The seeding (D3) is small: timed directly, the first seed_database() (create table + 2,000 rows) takes 0.120 s, and a start with a full table 0.0008 s.
  • total_*, chunks, node_modules. They jump with plotly.js, glide and gridjs. initial_gzip on / grows by 9 % (the navigation bar).
  • wire.hydrate and events.simple.capacity. Both carry the cost of rx.dynamic (next section).

rx.dynamic dominates the event numbers (kept on purpose). The ticket asks for rx.dynamic. Its var is cache=False and registered in _always_dirty_computed_vars, so reflex rebuilds the widget on every event of every state. I measured a closed loop of BenchState.set_seq, 10 sessions, disk manager, 2 runs each:

Appev/s
main's playground2,404, 2,444
this branch936, 871
this branch, only the 4 old pages2,201, 2,251
only the old pages + /widgets973, 959
+ /widgets without the one rx.dynamic call2,256, 2,165
full branch on the memory manager1,088, 1,131

I kept rx.dynamic (maintainer's decision) and documented its cost in the playground README. It also adds a frame to the hydration (on_load_internal sends the widget's 606-byte var before the is_hydrated delta).

6. Size budgets

The budgets are the measured values plus 5 % (20 % for node_modules), rounded up. Latest check, on a6acf6ee5: initial_gzip 340,728 B against a budget of 358,000; total_gzip 4,199,292 B against 4,410,000; 401 chunks against 422; node_modules 632,174,927 B against 759,000,000. reflex-bench budgets check exits 0.

7. D2: the frames the real frontend sends

Recorded from Chromium against prod on both versions (identical):

fresh browser, hydrate /:                                ['hydrate', 'on_load_internal']
fresh browser, navigate / -> /counter:                   ['on_load_internal']
after setting a localStorage var, navigate -> /counter:  ['update_vars_internal', 'on_load_internal']
after setting a localStorage var, reload /:              ['hydrate', 'update_vars_internal', 'on_load_internal']

The frontend sends update_vars_internal only when the browser already holds a value of a client storage var. Every benchmark session is a fresh browser, so WireSession and the priming in drivers/events.py already send what the frontend sends, and no harness code changed. Only the WireSession.navigate docstring and the README's "Wire sizes" wording changed.

8. Hash and repo checks

  • uv run python scripts/hash_examples.py --check passes.
  • uv run ruff check . and uv run ruff format --check . are clean.
  • uv run pyright reflex tests packages/reflex-bench/src examples: 0 errors.
  • uv run pytest tests/units/reflex_bench tests/units/test_testing.py -q passes.
  • uv run pre-commit run --files <changed files> is clean.
  • All 271 playwright integration tests pass locally.

News fragment: news/7359.bugfix.md, for the AppHarness fix below (the only change under reflex/).

Follow-up commits after opening

AppHarness state leak (reflex/testing.py, 204e7cd2d).

  • Symptom: every playwright integration test failed with KeyError: 'playground___states___widgets____widget_state'.
  • Cause: tests/integration/tests_playwright/test_playground.py runs the playground in-process. The harness only forgot the states of <app>.<app>, so the playground's WidgetState (in playground.states.widgets, with the always-dirty rx.dynamic var) reached the deltas of every later app in that process.
  • Fix: the harness now forgets the states of every <app>.* module when it stops.
  • Tests: tests/units/test_testing.py::test_app_harness_reload_forgets_states_of_every_app_module fails without the fix. All 271 playwright integration tests pass locally, and the integration gate is green.
  • Not changed: restarting a same-named multi-module app in one process doesn't re-register its cached submodules' states. That is also the case on main; see the review thread.

Review findings (a6acf6ee5).

  • Room names are derived server-side from the session, so no event can name or remove another visitor.
  • Each visitor gets an upload folder.
  • set_points and repeat are bounded.
  • One email check serves blur and submit.
  • The stats route is GET/HEAD only.
  • rx.video plays a real clip (assets/clip.webm, 7.4 kB).
  • New opt-in test_interactions_work: events, tasks, the data table and form, sign-ups, two-visitor uploads, storage, a two-visitor room and the stats route. It passes on HEAD and 0.8.23, and on redis.
  • The size budgets still pass: total_gzip 4,199,292 B, 401 chunks.

Notes for P2-1 (CI wiring)

  • The workspace subject needs the playground's packages in the harness environment: sqlmodel and alembic (uv sync --no-dev --extra db) and plotly. uv export --no-dev --extra db --package reflex does not include plotly, so a benchmark job that skips the dev group must also install plotly>=6.0,<7.0, for example uv pip install "plotly>=6.0,<7.0" after the sync.
  • size_budgets.yml and the head job of examples.yml sync the dev group, which has all three.

Owner actions

  • Decide whether Greptile's rule "extract string literals used as identifiers into constants" applies to the playground's documented element ids. I left them literal (see that review thread).

Deviations from the ticket

  • Branch. The session was pinned to farhan/funny-ptolemy-2t8klw, not farhan/eng-12606-playground-build-out.
  • Route name. /item/[item_id] instead of /item/[id], as the contract fixes.
  • rx.Model is deprecated at HEAD (0.9.2: "Directly use database ORM layer, like sqlalchemy or SQLModel"). The non-deprecated form, a plain sqlmodel.SQLModel, needs an import other than reflex, which the playground forbids. The ticket asks for rx.Model, so HEAD logs one deduplicated deprecation warning. rx.session() is not deprecated.
  • Other expected warnings. rx.dynamic logs "The _evaluate method is experimental". Reflex also warns "Database is not initialized" because the app creates its table with create_all instead of alembic migrations.
  • Theme. rx.App(theme=...) is a deprecated shortcut at HEAD, so the theme goes to RadixThemesPlugin(theme=...) in rxconfig.py (guarded like before). On 0.8.23, which has no plugin, playground.py sets app.theme.
  • Appearance toggle is on /settings, not in the shared navigation bar. On 0.8.23, a page rendering rx.color_mode.button stops applying its hot updates: hmr.render.leaf and hmr.watcher failed until it moved (bisected; with the button removed the edit shows after 3 s).
  • No ids on rx.foreach rows. An id built from a var gets a ref whose name embeds a random var id, so the build changed on every compile, and 0.8.23 stopped applying hot updates after the first. Rows have no ids; the README says why.
  • Plotly figure as a dict. rx.plotly gets a plain dict (ChartsState.figure.to(Any)), but the plotly package is still required: the component imports it.
  • Substates. 13 new substates against a target of 8–12.
  • rx.dynamic kept. It sets the level of every event benchmark (see above), by decision.
  • reflex/ change. The AppHarness fix touches reflex/testing.py and adds a news fragment. Without it, the playground's in-process integration test broke every later playwright test.

Follow-ups

  • A tailwind dim for lifecycle.*/size.*: PLAYGROUND_TAILWIND is not in any series key yet, and the README warns that setting it in the shell changes results without changing the key.
  • Framework, rx.dynamic: one always-dirty computed var in one substate taxes every event in the app (−57 % simple-event throughput here). Worth an issue.
  • Framework, prod server: it answers dynamic routes with status 404 (the page still renders), which logs a console error on every direct load of /item/42.
  • Framework, 0.8.23 (backfill only): rx.color_mode.button blocks hot updates of its page, and var-built ids inside rx.foreach make compiles non-deterministic. Check whether HEAD still has the second.
  • Framework, AppHarness: restarting a same-named multi-module app in one process loses its submodules' states (also on main).
  • Environment: hmr.css and 0.8.23's hmr.render.root and memory.leak fail on main on this VM too; confirm on the macro runner before gating them.

Adversarial self-review

  1. The fixture's cost is dominated by one widget (rx.dynamic). Kept by decision; documented in the playground README and above.
  2. Harness test edit. test_wire_app.py now hardcodes 5 hydration frames. That is a fact of the fixture, like the old 4, and the comment names each frame.
  3. Blocking DB call in async code. api.py runs a synchronous sqlite count inside the ASGI coroutine. It is a small query on a local file, but it blocks the event loop for its duration. rx.asession would need the aiosqlite driver, which is not installed.
  4. Backend-only rows per session. AnalyticsState._rows holds all 2,000 rows per session that opens /data/analytics, and the disk and redis managers pickle them on each of that state's events. That is the intended "expensive" case, and no benchmark visits the page.
  5. Upload paths. Uploads go to a random per-visitor folder, and names are reduced to their base name, skipping ./...
  6. seed_database uses sqlite-specific SQL (INSERT OR IGNORE, exec_driver_sql). db_url is sqlite by configuration.
  7. Evidence hash. The content hash moved after the benchmark runs (README note, then the review fixes), so the committed hash differs from the one recorded in the benchmark results above. The console sweep, interactions and size budgets were re-run on the final code.
  8. Targeted pyright ignores. Seven: four for sqlmodel/sqlalchemy column and constructor typing, the data editor and upload event specs, and a color_scheme Literal. Each is a stub limitation, not a runtime problem.
  9. Attribution lines. The commits end with a Claude-Session link line, following this environment's commit convention. The task asked for no Claude attribution; if that link counts, the lines can be dropped when squashing.

https://claude.ai/code/session_01Y9bdwnG6ar7T4ay9nWmKb5