improvement(desktop-browser): stable element refs, wait observations, and fewer false refusals
Merged
WA
2d ago+523+523+012345678901234567890123456789-79-79-01234567890123456789
Summary
- Element refs stay stable: an unchanged element keeps its ref across snapshots, and a ref from an earlier snapshot still resolves while its element lives (previously any later snapshot, including query/scoped observations, invalidated every ref the model held)
- Text lines register the identity name the resolver recomputes — aria-labelled text lines and lines over 120 characters were never actionable by ref
- A still-mounted control whose own menu/disclosure opened keeps its ref instead of reporting "identity drifted" (hover menus failed right after a successful hover)
browser_wait_forends with a fresh snapshot, so the model doesn't spend a round re-observingbrowser_typetypes into a top-page field covered by a non-modal overlay (keyboard focus, as Playwrightfilldoes); modal backgrounds and framed fields still refuse- One bounded ref store per page (5000) and per driver scope (LRU), shared frame/target helpers
Type of Change
- Improvement
Testing
- Red/green unit tests for each change (
page-functions.test.ts,driver.test.ts,post-action-observation.test.ts); 534 browser-agent tests pass - Lint, type-check, block registry, generators clean;
check:auditsonly failed knip on the docs app's generated.contentimport, andbun run testonly failed two desktop tests that also fail on staging when run as root (permission-refusal tests) - Production-loop benchmark (worker main agent, Opus 5.5 high, Sim's Electron driver; capbench 56 + WebGames 104 + REAL 121 paired episodes, fresh seeds, arms run concurrently): success 214 → 215, rounds/episode −12%, failed tool calls −36%, cost −9%, wall −7%
Checklist
- Code follows project style guidelines
- Self-reviewed my changes
- Tests added/updated and passing (new tests pass the
test-auditauthoring gate) - No new warnings introduced
- I confirm that I have read and agree to the terms outlined in the Contributor License Agreement (CLA)