Fix recording pipeline and deterministic replay on browser-use 0.13 / Chrome 137+

SA
2mo ago
+449-76

Fixes #165

On a fresh clone with the pinned browser-use 0.13.4, recording captures zero events and deterministic replay fails on several step types. This PR restores the full record → build → replay (no-AI) pipeline; each commit is self-contained:

1. fix(recorder) — recorder extension never loads

  • browser-use's default extensions append a second --load-extension flag which overrides ours (Chrome honors the last one). Disable default extensions for the recording profile.
  • Branded Google Chrome 137+ removed --load-extension support altogether. Prefer a Playwright-bundled Chromium / Chrome for Testing binary when present (macOS/Linux/Windows cache paths), overridable via WORKFLOW_USE_RECORDER_BROWSER; falls back to browser-use's default with a warning.

2. fix(schema) — validator 422s every incremental recording update

validate_ends_with_extract rejected essentially every WORKFLOW_UPDATE streamed by the extension mid-recording, so the recording server never stored any data and create-workflow hung after the browser closed. The validator is relaxed; no-AI workflows are also valid without a trailing extract step.

3. fix(workflow) — deterministic replay on browser-use 0.13

  • Stop stripping cssSelector/xpath from step params: the click/input/key_press/select_change action models require cssSelector (extras are ignored via extra='ignore'), so every selector-based deterministic action failed validation.
  • Named-key crash: dict.get's default is evaluated eagerly, so ord('ENTER') raised even though 'Enter' is in the key map.
  • Replace the hand-rolled Input.dispatchKeyEvent (stale CDP session → -32601 method not found) with page.press().
  • Key-press verification treated navigation as failure (element gone → retry pressing Enter repeatedly). A URL change now counts as success.
  • Retry semantic-mapping extraction briefly when it comes back empty (page mid-navigation) instead of failing the step.

4. fix(cli,backend) — no-AI paths shouldn't require BROWSER_USE_API_KEY

  • cli.py: NameError: llm_instance when LLM init failed; now degrades gracefully.
  • backend: from browser_use.browser.browser import Browser no longer exists in 0.13; and the eager ChatBrowserUse instantiation made every endpoint 500 without an API key. LLM is now optional.

Verified

  • Recording: extension service worker present on the CDP debug port; step events received by the server; create-workflow-no-ai produced a valid semantic workflow from a real user recording.
  • Replay: run executes navigation/input/key_press/click/scroll steps of the recorded workflow deterministically (no API key), including Enter-triggered navigation.

🤖 Generated with Claude Code

Summary by cubic

Restores the full record → build → deterministic replay pipeline on browser-use 0.13, adds a GUI “Record New Workflow” button backed by new recording APIs, and hardens task/recording state so status/cancel work across requests and concurrent runs. No‑AI workflows now run without BROWSER_USE_API_KEY.

  • New Features

    • GUI recording: sidebar button starts/stops recording, polls /recordings/status, and selects the saved file when done.
    • Backend recording: /recordings/start|stop|status run RecordingService, convert to semantic, and save into ./tmp; zero‑step stops cancel cleanly.

  • Bug Fixes

    • Recorder: disable default extensions so our --load-extension wins; prefer Playwright-bundled Chromium/Chrome for Testing (newest revision) on Chrome 137+; override via WORKFLOW_USE_RECORDER_BROWSER.
    • Backend: share one WorkflowService to preserve in‑memory task/recording state; create Browser/WorkflowController per execution; prune finished tasks; record load errors in task status.
    • Schema: relax “must end with extract” so incremental recording updates are accepted.
    • Workflow: keep cssSelector/xpath; use page.press() for keys; treat URL change as success; retry semantic mapping during navigation.
    • CLI/Backend: optional LLM; run-workflow guards on missing LLM; update imports for browser-use 0.13.
    • UI: use GET /api/workflows/metadata so the sidebar shows real names; keep metadata cache in sync after saves; record button stop/polling logic is more robust.