improvement(desktop-browser): stable element refs, wait observations, and fewer false refusals

Merged
WA
2d ago
+523-79

Summary

  • Element refs stay stable: an unchanged element keeps its ref across snapshots, and a ref from an earlier snapshot still resolves while its element lives (previously any later snapshot, including query/scoped observations, invalidated every ref the model held)
  • Text lines register the identity name the resolver recomputes — aria-labelled text lines and lines over 120 characters were never actionable by ref
  • A still-mounted control whose own menu/disclosure opened keeps its ref instead of reporting "identity drifted" (hover menus failed right after a successful hover)
  • browser_wait_for ends with a fresh snapshot, so the model doesn't spend a round re-observing
  • browser_type types into a top-page field covered by a non-modal overlay (keyboard focus, as Playwright fill does); modal backgrounds and framed fields still refuse
  • One bounded ref store per page (5000) and per driver scope (LRU), shared frame/target helpers

Type of Change

  • Improvement

Testing

  • Red/green unit tests for each change (page-functions.test.ts, driver.test.ts, post-action-observation.test.ts); 534 browser-agent tests pass
  • Lint, type-check, block registry, generators clean; check:audits only failed knip on the docs app's generated .content import, and bun run test only failed two desktop tests that also fail on staging when run as root (permission-refusal tests)
  • Production-loop benchmark (worker main agent, Opus 5.5 high, Sim's Electron driver; capbench 56 + WebGames 104 + REAL 121 paired episodes, fresh seeds, arms run concurrently): success 214 → 215, rounds/episode −12%, failed tool calls −36%, cost −9%, wall −7%

Checklist

  • Code follows project style guidelines
  • Self-reviewed my changes
  • Tests added/updated and passing (new tests pass the test-audit authoring gate)
  • No new warnings introduced
  • I confirm that I have read and agree to the terms outlined in the Contributor License Agreement (CLA)