fix(agent): show the judge the current location

Merged
TW
12d ago
+299-34

What and why

  • The built-in judge never saw the current URL, so agent.assert and agent.waitFor about the address ended ASSERTION_INCONCLUSIVE or STEP_TIMEOUT, whether or not it matched. agent.act already reads the location.
  • buildPrompt now puts it on <observation> (packages/e2e/src/agent/prompts.ts:101). It is the same redacted value act reads, a path on the app's origin and the whole address off it, without userinfo, cut at 256 characters like a link target (judgedLocation in observation.ts).
  • JUDGMENT_REQUEST gets one line: the location is evidence only for an instruction about the address, never for what the screen shows.
  • waitFor judges again when only the location changes, as after history.replaceState (index.ts:390). It skips that under vision: 'only', which is not shown the location.
<observation revision="b1" location="/page" viewport="1280x720">

Issue

Fixes #1514

Verified

Ran it locally: yes

Testmainthis branch
control, toHaveURL('/page')passespasses
assert A, The URL path is /pageASSERTION_INCONCLUSIVEpasses
assert B, The URL path is /otherASSERTION_INCONCLUSIVEASSERTION_FAILED as expected
waitFor A, The URL path is /pageSTEP_TIMEOUTpasses
waitFor B, only the URL changesSTEP_TIMEOUTpasses
  • Web benchmark tests-agent, --strict-cache with gpt-6-luna in place of the gateway model: 33 passed, no REPLAY_STALE.
  • pnpm bench:ab on the web benchmark: every test ran the same steps with the same outcome on both builds, and suite time was same.
  • Not run: --no-cache --ai-trace on the agent suites.

Checklist

  • Changeset added (pnpm changeset) for any change to a published package, or the change is testbed, docs, or CI only.
  • Breaking change: title carries !, the changeset body names what breaks and what replaces it, and the deprecation warning landed a window earlier (stability policy in CONTRIBUTING.md).
  • Docs updated in the same PR: the docs/**/*.mdx page for the behavior, and skills/e2e/ if the skill describes it.
  • Contract change: emitted .d.ts reviewed, tests/types/sdk-types.ts updated; wire change edits the schema and both fixtures.
  • Engine contract change (e2e/engine): changesets for e2e, @e2e-dev/web, and @e2e-dev/mobile.

šŸ¤– Generated with Claude Code