Browser tests for every docs demo, written beside the demo

MA
2d ago
+9062-110

Every interactive demo in the docs now has a browser test, written in the markdown directly below the demo it exercises:

```python demo
rx.button("Increment", on_click=State.increment)
```

```python test
def test_counter(demo):
    demo.get_by_role("button", name="Increment").click()
    expect(demo.get_by_role("heading")).to_have_text("1")
```

How it works

  • reflex-site-shared pairs each python demo / demo-only / eval block with the python test blocks after it (iter_doc_demos). Test blocks are not rendered and are stripped from the markdown served to agents. Each rendered demo is wrapped in a display: contents div carrying data-docs-demo=<index of the demo in its document>.
  • docs/app/tests/demos/conftest.py collects one pytest module per tested demo, grouped by route (tests/demos/test_doc_demos.py::/docs/library/forms/button/::L42). The code is compiled against the markdown file at its own line numbers, with asserts rewritten, so tracebacks point into the docs. Tests use reflex-enterprise's reflex_app harness and get two fixtures: page (at the route, hydrated) and demo (a locator for the rendered demo only).
  • Only the routes under test are compiled (REFLEX_DOCS_WHITELIST). The app gets OS-assigned free ports, and REFLEX_PERSIST_LOCKFILES=false keeps the run from rewriting the committed reflex.lock/.
  • A demo with nothing worth testing (a screenshot, a spacer, a fixed table) is flagged notest (```python eval notest). --docs-demos-untested lists demos that have neither tests nor the flag; there are none.
  • CI: a new docs-demos job in docs_tests.yml runs across 5 shards by docs section, on changes to the docs or the framework. The last two shards select whatever the earlier ones leave, so a new section is never skipped. The job is part of docs-tests-gate. It runs in dev mode, because reflex-enterprise runs --env prod only for a logged-in Reflex account.

Coverage

697 demos have tests and 84 are notest. Locally, in both prod and dev: 705 passed, 12 xfailed, 3 skipped. The skips need external services. The slowest shard takes about 5 minutes.

The 12 strict xfails are real bugs the tests found, left for follow-ups:

  • String vars: .to_string() adds JSON quotes, and .split() with no separator splits into characters.
  • The browser_storage settings dialog crashes with a FormLabel error.
  • The rx.checkbox label never renders.
  • Recharts: bar_size has no effect, and the funnel chart legend is empty.
  • AG Grid shows no selection checkboxes, and its cell-change handler reads fields the event doesn't send.
  • The map zoom handler reads _zoom instead of zoom.
  • Two styling claims in the docs don't hold.
  • The dashboard tutorial's dialogs stack, because its demos share state.

Framework changes

  • REFLEX_PERSIST_LOCKFILES (new, default true): set it to false and a frontend install no longer writes back to the app's reflex.lock/. reflex now requires reflex-base >= 0.11.1.dev0, the version that adds it.
  • Bugfix: reflex run --env prod and preview mode printed the app URL without frontend_path, so following it gave a 404. A regression test covers it.
  • The docs article's padding reset now reaches through the demo wrapper ([&>[data-docs-demo]>div]:!p-0). Without it, six dashboard tutorial blocks rendered with 32px of extra padding.

Also noticed, not fixed here

  • The pyplot demo's computed vars render matplotlib on every docs page hydrate, in production too.
  • The match.md prose contradicts its demo code.

🤖 Generated with Claude Code