fix(agent): show the judge the current location
Merged
TW
12d ago+299+299+012345678901234567890123456789-34-34-01234567890123456789
What and why
- The built-in judge never saw the current URL, so
agent.assertandagent.waitForabout the address endedASSERTION_INCONCLUSIVEorSTEP_TIMEOUT, whether or not it matched.agent.actalready reads the location. buildPromptnow puts it on<observation>(packages/e2e/src/agent/prompts.ts:101). It is the same redacted valueactreads, a path on the app's origin and the whole address off it, without userinfo, cut at 256 characters like a link target (judgedLocationinobservation.ts).JUDGMENT_REQUESTgets one line: the location is evidence only for an instruction about the address, never for what the screen shows.waitForjudges again when only the location changes, as afterhistory.replaceState(index.ts:390). It skips that undervision: 'only', which is not shown the location.
<observation revision="b1" location="/page" viewport="1280x720">
Issue
Fixes #1514
Verified
Ran it locally: yes
- The issue's reproduction, https://github.com/two-pack/e2e-judge-url-repro, run with
npm testandgpt-6-luna:
| Test | main | this branch |
|---|---|---|
control, toHaveURL('/page') | passes | passes |
assert A, The URL path is /page | ASSERTION_INCONCLUSIVE | passes |
assert B, The URL path is /other | ASSERTION_INCONCLUSIVE | ASSERTION_FAILED as expected |
waitFor A, The URL path is /page | STEP_TIMEOUT | passes |
| waitFor B, only the URL changes | STEP_TIMEOUT | passes |
- Web benchmark
tests-agent,--strict-cachewithgpt-6-lunain place of the gateway model: 33 passed, noREPLAY_STALE. pnpm bench:abon the web benchmark: every test ran the same steps with the same outcome on both builds, and suite time wassame.- Not run:
--no-cache --ai-traceon the agent suites.
Checklist
- Changeset added (
pnpm changeset) for any change to a published package, or the change is testbed, docs, or CI only. - Breaking change: title carries
!, the changeset body names what breaks and what replaces it, and the deprecation warning landed a window earlier (stability policy in CONTRIBUTING.md). - Docs updated in the same PR: the
docs/**/*.mdxpage for the behavior, andskills/e2e/if the skill describes it. - Contract change: emitted
.d.tsreviewed,tests/types/sdk-types.tsupdated; wire change edits the schema and both fixtures. - Engine contract change (
e2e/engine): changesets fore2e,@e2e-dev/web, and@e2e-dev/mobile.
š¤ Generated with Claude Code