Browser tests for every docs demo, written beside the demo
MA
2d ago+9062+9062+0123456789012345678901234567890123456789-110-110-012345678901234567890123456789
Every interactive demo in the docs now has a browser test, written in the markdown directly below the demo it exercises:
```python demo
rx.button("Increment", on_click=State.increment)
```
```python test
def test_counter(demo):
demo.get_by_role("button", name="Increment").click()
expect(demo.get_by_role("heading")).to_have_text("1")
```
How it works
- reflex-site-shared pairs each
python demo/demo-only/evalblock with thepython testblocks after it (iter_doc_demos). Test blocks are not rendered and are stripped from the markdown served to agents. Each rendered demo is wrapped in adisplay: contentsdiv carryingdata-docs-demo=<index of the demo in its document>. docs/app/tests/demos/conftest.pycollects one pytest module per tested demo, grouped by route (tests/demos/test_doc_demos.py::/docs/library/forms/button/::L42). The code is compiled against the markdown file at its own line numbers, with asserts rewritten, so tracebacks point into the docs. Tests use reflex-enterprise'sreflex_appharness and get two fixtures:page(at the route, hydrated) anddemo(a locator for the rendered demo only).- Only the routes under test are compiled (
REFLEX_DOCS_WHITELIST). The app gets OS-assigned free ports, andREFLEX_PERSIST_LOCKFILES=falsekeeps the run from rewriting the committedreflex.lock/. - A demo with nothing worth testing (a screenshot, a spacer, a fixed table) is flagged
notest(```python eval notest).--docs-demos-untestedlists demos that have neither tests nor the flag; there are none. - CI: a new
docs-demosjob indocs_tests.ymlruns across 5 shards by docs section, on changes to the docs or the framework. The last two shards select whatever the earlier ones leave, so a new section is never skipped. The job is part ofdocs-tests-gate. It runs in dev mode, because reflex-enterprise runs--env prodonly for a logged-in Reflex account.
Coverage
697 demos have tests and 84 are notest. Locally, in both prod and dev: 705 passed, 12 xfailed, 3 skipped. The skips need external services. The slowest shard takes about 5 minutes.
The 12 strict xfails are real bugs the tests found, left for follow-ups:
- String vars:
.to_string()adds JSON quotes, and.split()with no separator splits into characters. - The
browser_storagesettings dialog crashes with aFormLabelerror. - The
rx.checkboxlabel never renders. - Recharts:
bar_sizehas no effect, and the funnel chart legend is empty. - AG Grid shows no selection checkboxes, and its cell-change handler reads fields the event doesn't send.
- The map zoom handler reads
_zoominstead ofzoom. - Two styling claims in the docs don't hold.
- The dashboard tutorial's dialogs stack, because its demos share state.
Framework changes
REFLEX_PERSIST_LOCKFILES(new, defaulttrue): set it tofalseand a frontend install no longer writes back to the app'sreflex.lock/. reflex now requiresreflex-base >= 0.11.1.dev0, the version that adds it.- Bugfix:
reflex run --env prodand preview mode printed the app URL withoutfrontend_path, so following it gave a 404. A regression test covers it. - The docs article's padding reset now reaches through the demo wrapper (
[&>[data-docs-demo]>div]:!p-0). Without it, six dashboard tutorial blocks rendered with 32px of extra padding.
Also noticed, not fixed here
- The pyplot demo's computed vars render matplotlib on every docs page hydrate, in production too.
- The
match.mdprose contradicts its demo code.
🤖 Generated with Claude Code