#231 browser-use: run the agent in Rust (stacked on #166)

Stacked on #166. This replaces the Python agent in browser/ with Rust under src/browser/. wally browser-use now runs with just the binary and Chrome: no uv, no Python environment built on first run, no browser-harness.

The loop is still Jev Ultrafast's. It reads the page with snapshot.js and asks one /systemone question set for the next operation and its target. The guards run before every action, and a decision made on a page that has changed since is thrown away.

What changed

• src/browser/: the loop, a DevTools client over tungstenite, the page snapshot, the decision request, the text helper and the guards.
• contracts/: the generator can now emit the createSystemOne binding (src/account/systemone_contract.rs), and sync_from_inferenceinfra.py --check checks it alongside the other two bindings. Both existing bindings regenerate byte-identical.
• Request bodies are built in insertion order. Sorting the keys changed jev's scores on a recorded Google Flights request: the click target's top probability fell from 0.71 to 0.47.
• The page reader now handles the cases that failed on real sites:
  • native checkboxes, radios and selects hidden under drawn controls (Google Flights' Stops dialog, Amazon's Sort by);
  • open shadow roots and same-origin iframes;
  • divs and spans used as buttons.
• Cross-origin iframes stay unreadable on purpose, which keeps card fields out of reach.
• When a large page is split across several questions and every group answers "none", jev now gets a final round among each group's best control instead of the run stopping.
• --chrome attach without --cdp-url connects to the person's own Chrome through chrome://inspect/#remote-debugging. Chrome 136 and later ignore --remote-debugging-port on the default profile.
• Flags: --start-url is required. --max-steps and --chrome attach work. --profile and --clean-output are removed because nothing read them.
• The default text model is now deepseek-v4.1-flash-fast: a 612 ms median against 1,449 ms for deepseek-v4.1-flash on our gateway, both 5/5 correct.

Tests

• 75 unit tests in src/browser/. They include the Python tests ported one for one and guard checks over four captured payment pages (Stripe, Braintree).
• tests/test_wally_browser.rs: 26 tests running the real binary against headless Chrome, a local site and a fake model that always reaches for the most dangerous control:
  • Pay is never pressed, no card field is typed into, and a confirm dialog is never accepted;
  • 15 page patterns, among them a skinned checkbox, radio and select, a listbox, an autocomplete list, a date grid, a modal and a below-the-fold button;
  • also iframes, shadow DOM, div buttons, the final round, --max-steps, attach mode and a gateway 503.
• These tests are skipped, with a printed reason, on machines without Chrome.
• ctest, clippy -D warnings, fmt, the test ledger, the contract check and the agents sync all pass locally.

Live runs with jev-latest on the Experiential gateway (eve on our gateway is returning 503):

Task	Result
Wikipedia, open Bengaluru	Done, about 7 s
Google Flights, one-way, 2 adults, nonstop	Done, about 30 s
Amazon, sort by price and add an SSD to the cart	Done, about 23 s

Site coverage

Tested live with jev-latest on the Experiential gateway:

Site	Task	Result
Wikipedia	Search, open an article	Done
Google Flights	One-way, 2 adults, a date, the Nonstop filter	Done
Amazon.in	Search, sort by price, open a result, add to cart	Done
Amazon.in	"Buy it now"	Stops on the product page; Buy Now is never pressed

The local suite also covers 18 ways sites build controls:

• hidden native checkboxes, radios and selects;
• custom listboxes, autocomplete lists and date grids;
• modals, controls below the fold, buttons that start disabled, and contenteditable;
• same-origin iframes and open shadow roots, including a shadow root inside an iframe;
• div buttons;
• new tabs opened by a link, by a noopener link, and by script.

It also has seven probes that must never pay or type into a card field.

Complex sites aren't supported yet. What doesn't work today:

• Controls in cross-origin iframes. This is on purpose: it keeps card fields out of reach, but embedded third-party widgets can't be used either.
• Closed shadow roots, canvas-drawn UIs, file uploads, native <input type=date>, drag-and-drop and keyboard-only widgets.
• Logins with a password. The agent never types into password fields, so the person signs in first in the Wally Chrome profile, or uses --chrome attach.
• Large pages, which get split into several questions: eve takes 26 options per question and jev-latest on Experiential takes 64. Splitting costs accuracy and speed.
• The payment and commit guard matches English keywords.

What full support would take:

• Raising the per-question option cap on our gateways towards TypeSafe's 255, so a page goes in one question.
• Driving cross-origin frames through Chrome's per-frame DevTools sessions, with the card-field guards applied inside each frame.
• Input support for uploads, native date inputs and drag-and-drop.
• A guard that reads the page's own structure (payment autocomplete fields, checkout forms) rather than only labels, so it works in any language.
• A recorded corpus of real-site tasks run in CI, so a regression on a real site shows up before release.

Summary by cubic

wally browser-use now runs entirely in Rust, replacing the Python agent under browser/ with src/browser/. The command needs only the binary and Chrome—no uv, Python environment, or browser-harness. The agent loop stays Jev Ultrafast's, but the page reader now handles skinned checkboxes/radios/selects, open shadow roots, same-origin iframes, and div/span buttons. Cross-origin iframes and closed shadow roots remain unreadable, keeping card fields out of reach.

Behavior

• --start-url is required; --max-steps and --chrome attach work. --profile and --clean-output are removed because nothing read them.
• Request bodies are built in insertion order; sorting keys changed model scores.
• When a large page splits and every group answers "none", a final round picks among each group's best control.
• A createSystemOne contract binding is generated and checked alongside the existing two.
• The default text model is now deepseek-v4.1-flash-fast.
• On money pages, buttons whose label commits an order, booking, or transfer are refused, so "Proceed to checkout" no longer depends on the goal text.
• Text is typed only where focus is confirmed safe after a click; a cross-origin frame focus is refused. The field's current value is selected in code, so no select-all key event reaches page handlers that could move focus.
• A target already used twice in recent history is held back.
• Clicks that open a new tab are followed.

Testing

• 75 unit tests in src/browser/, including the Python tests ported one for one.
• 26 end-to-end tests run the real binary against headless Chrome and a fake model that always reaches for the most dangerous control.
• Tests cover payment guards, skinned controls, iframes, shadow DOM, div buttons, the final round, --max-steps, attach mode, and gateway errors.
• Tests skip with a printed reason when Chrome isn't installed.