browser-use: complex-site support, part 1 (stacked on #166)

Merged
SI
2d ago
+563-12

Stacked on #166. This is the first step of complex-site support, for jev-latest on the Experiential gateway. Eve is out of scope.

What changed

  • Native date and time inputs (date, time, month, week, datetime-local) are now offered as fields. The text model writes the value in the input's own format, the format is checked in Rust, and the value is set in code. Typed keys don't reach these pickers.
  • Payment forms in any language. The snapshot records which buttons submit a form, and whether that form holds a card, one-time-code or password field. Submitting such a form is refused whatever the button says. A commit-word control is also refused while a card field is on screen. A German checkout ("Zahlungspflichtig bestellen") is covered.
  • Word rules apply to button-length labels only (8 words or fewer). Flipkart product titles such as "… Fast Charge …" or "data transfer cable" were being refused as payment controls.
  • Inner scrolling. On app-style pages where the document doesn't scroll (Flipkart's product page scrolls a 5,379 px inner div), the largest scroll container gets the scroll actions, and the wheel goes over its middle.
  • Late-loading tabs. A followed tab is read only after it leaves about:blank. Flipkart opens products in a new tab that starts blank.
  • Every guard stop gives its reason, for example refused "Buy Now": it would pay, place an order, or send a card or password.

Tests

  • 104 unit tests and 41 integration tests against headless Chrome. The new ones cover a date input, a date-time input, the German checkout probe, an inner scroller and a late-loading tab.
  • The German probe fails under the old word-only guard and passes with the structural rule.
  • ctest, clippy -D warnings and fmt are clean.

Live on a Mac with jev-latest:

TaskResult
Wikipedia, open an articleDone
Amazon, sort and add to cartDone
Amazon "buy it now"Stopped: refused "Buy Now"
Flipkart, search, sort, open the productDone
Flipkart "buy it now"Never pays. It loops until the 60-action budget, because the only purchase control is an unmarked div
Google Flights, one-way, 2 adults, nonstopDone in 4 of the last 6 runs. The other 2 hit the 60-action budget with jev looping on the search form, the model's variance rather than a guard stop

Not yet

  • Flipkart's product buttons are bare divs with no role, no pointer cursor and no handler the page exposes, so the agent can't see "Add to cart" there.
  • Cross-origin iframes and file uploads come next. That's items 4 and 5 in the plan.

Summary by cubic

Adds the first batch of complex-site support for jev-latest: native date and time inputs are fillable, payment forms are refused by page structure in any language, app-style pages with inner scroll containers scroll correctly, late-loading tabs are read after they load, and every guard stop reports its reason.

Native inputs

  • date, time, month, week, and datetime-local fields are now offered; the model writes the value in the field's own format, Rust validates it, and the value is set in code because these pickers take no typed keys.
  • The field context tells the model the expected format; a badly formatted value fails with a clear error and types nothing.
  • A date or month fill is refused when it would be a card expiry: the label names one ("Valid thru" and friends), the form holds a card field, or a card field is visible anywhere on the page.

Navigation and guards

  • The snapshot marks buttons that submit a form and whether that form holds a card, one-time-code, or password field; submitting such a form is refused whatever the button says, covering a German checkout.
  • Pay and place-order wording is judged at any label length, so long saved-card pay buttons are refused too; softer commit words judge button-length labels only, so long product titles like Flipkart's are no longer refused.
  • Button-shaped boxes with no role, cursor, or handler (Flipkart's div "Add to cart") are offered as click targets.
  • Scroll actions target the largest inner scroll container when the document does not scroll; followed tabs are read only after leaving about:blank.
  • Every guard stop prints why, e.g. refused "Buy Now".

The remaining hard sites are cross-origin iframes and file uploads; those come next.