feat(e2e): put the test title in agent context

Merged
KA
5d ago
+90-30

Under e2e run, the agent now knows which test it is in. Every prompt's context opens with the test's title path:

Project context:
Test: billing > upgrades the tier
Plans are called tiers.                    <- agents.<name>.context
The seeded workspace is "Acme Trial".      <- agentContext
  • The act loop, the judges behind assert, waitFor, and extract, and a custom executor's ctx.agentContext all get the line (packages/e2e/src/run/fixtures.ts). It is always on, with no opt-out.
  • The replay cache key leaves the title out. The key's test id is already built from file and title path, so renaming a test still re-records, and the committed benchmark recordings keep replaying. AgentSelection now carries two fields: agentContext, what the model reads, and keyedContext, what the key digests (packages/e2e/src/agent/act.ts).
  • Titles are redacted at collection and the context is redacted again per step, so a secret in a title reaches the model as its marker (tests/integration/plain-string-secrets.test.ts).
  • e2e explore drops its Exploration goal: <goal> line, since the goal is the test's title and would otherwise appear twice.
  • Because judges read the title, the docs and the skill say to name a test for what it covers, not for the verdict it expects.

Not covered: openHost sessions and e2e mcp have no e2e test, so their prompts get no title line. Docs scope the behavior to e2e run.

Verified

Ran it locally: yes

  • web benchmark, CI=1 pnpm --filter @e2e-dev/web-benchmark run test:agent --strict-cache against the built packages: 26 replayed, 0 REPLAY_STALE. The 7 failures are steps that call the model on any build: four have run-time values the recording can't replay, two have no committed recording, and one is a judgment. I ran without a model key, so they ended in MODEL_PROVIDER_FAILED. CI runs them with the key.
  • testbed default suite on the built CLI: 79 passed, 1 skipped (run on port 4371 from a temporary config because another worktree held 4271).
  • New unit tests show the title as the first context line in an act prompt and in a judgment's <project-context>. The trace-cache test checks that the recorded agentContextDigest for a test with no context is still the empty-string digest.
  • Not run: the mobile benchmark. Its context goes through the same fixtures.ts path.