docs(skills): rerun one test with the replay cache on while iterating
A coding agent fixing a red e2e test reruns the whole file over and over, and the docs told it to add --no-cache after every change. That makes every agent.act before the broken line call the model again on each iteration, even though the replay cache already keeps the goals a passing check confirmed before the failure. This gives the agent a fast inner loop instead.
skills/e2e/SKILL.md: writing step 4 runs one test byfile:line, and step 5 of "Workflow: fixing a failed test" (#1309) reruns it the same way instead of rerunning the file. A new section after that workflow, "Rerunning a test while you fix it", covers the loop:--action-timeout 5000(#1310), so a wrong locator fails in 5 s instead of 30 s; dropped for the final--no-cacherun- keep the cache on; what replays and what runs live
- which edits re-record a goal: renaming the test, its
describe, or its file; rewording a goal or its params; changing the agent'scontextor the test'sagentContext --retries 0when retries are on, since a retry never replays outside strict mode- a stopped run saves nothing
--no-cacheone time once the test passes
skills/e2e/references/agent.md, "The replay cache": what a failing run keeps and evicts, the full list of key parts, and when--no-cacheis the right call.skills/e2e/references/debugging.md: the--no-cacherow now says when to use it. Rerun with the cache on first, because the failed run already deleted the replay it implicates. If the failure repeats after a kept replay and--no-cachepasses, tighten the check after that goal (--no-cachewrites nothing, so the stale entry would otherwise replay forever). The "Read the failure" bullet from #1309 reruns byfile:line.skills/e2e/references/mcp.md: workflow step 4 runs and reruns that one test byfile:line.skills/e2e/references/running.md: the--retriesrow says a retry runs agent steps live.docs/coding-agents.mdx: fixes the "adds--no-cacheafter it changes a test" line and showsnpx e2e run tests/checkout.e2e.ts:3 --action-timeout 5000.docs/cache.mdx: a new "Rerun a test while you fix it" section. "What must match" now names moving the file and renaming adescribe.- A changeset (
e2e: patch), since the skill ships in the package.
Decisions:
- The loop is its own section in
SKILL.md, right after #1309's fixing workflow. That workflow says how to fix from the run's evidence; this section says how to rerun. The workflow's step 5 points here instead of repeating the flags. --action-timeoutgoes in the loop itself. In #1310's A/B runs, both agents read its tip in debugging.md's Tools table before their first run, and neither used the flag on it. The timeout isn't part of a recording's key (packages/e2e/src/cache/context.ts), so changing it keeps the recordings.- For a stale replay, the skill tells the agent to tighten the check, not to delete the entry under
.e2e/cache/. The skill's rule is that.e2e/is read-only, ande2e cache clearwould wipe every other test's recordings, committed ones included.
Docs shortening pass (authoring-docs): I cut no facts. The coding-agents paragraph replaces the wrong --no-cache sentence with the loop. Measured against current main:
| Page | Lines before → after | Prose words before → after |
|---|---|---|
docs/cache.mdx | 400 → 422 | 2646 → 2830 |
docs/coding-agents.mdx | 264 → 272 | 1055 → 1120 |
Follow-up cuts for cache.mdx, not done here: the --no-cache CodeGroup under "Why a step was not replayed" could become a link to the new section, and "Commit the replay cache" repeats the .gitignore fact from "Modes".
Verified
Ran it locally: yes, with a scripted model. No live model was reachable on this machine: the stored ChatGPT login returns 401 ("session has ended"), there are no other e2e login entries, and there is no API key. So the scratch project's agents.default.model is a scripted AI SDK v4 model. It reads node ids off the act-loop prompt and calls type, tap, and complete_step, and it logs every call. Everything else is the built CLI from this branch: runner, cache, @e2e-dev/web, Chromium, and the testbed playground app. The model's identity isn't part of the cache key, so the cache decisions below don't depend on which model answers. A real model only makes each live call slower.
Scratch test (testbed /forms and /wizard), run each time as npx e2e run tests/journey.e2e.ts:4 --debug:
test('a member saves a profile and creates a workspace', async ({ app, agent, screen }) => {
await app.open('/forms');
await agent.act('save the profile with the full name "Grace Hopper"');
await expect(screen.getByRole('status', 'Save result')).toHaveText('Saved profile for Grace Hopper');
await app.open('/wizard');
await agent.act('name the workspace "Atlas" and go to the next step');
await expect(screen.getByLabel('Plan')).toBeVisible();
await screen.getByLabel('Plan').selectOption('Pro');
await screen.getByRole('button', 'Next').tap();
await screen.getByRole('button', 'Create workspace').tap();
await expect(screen.getByRole('status', 'Summary')).toHaveText('Created "Atlas" on the Pro plan');
});
step.cache of the two agent.act steps, from .e2e/report.json, as mode · write:
| Run | Change | Result | Goal 1 | Goal 2 | Model calls | Duration |
|---|---|---|---|---|---|---|
| 1 | new test | pass | missed · saved | missed · saved | 6 | 1.8 s |
| 2 | later line broken: 'Create the workspace' | LOCATOR_NOT_FOUND | self-finalized · kept | self-finalized · kept | 0 | 31.2 s |
| 2b | same, --retries 1 | fail, 2 attempts | attempt 1 replayed and kept, retry missed (retry) | same | 6 | 62.5 s |
| 3 | line fixed | pass | self-finalized · kept | self-finalized · kept | 0 | 1.4 s |
| 4 | goal 2 reworded | pass | self-finalized · kept | missed · saved | 3 | 1.7 s |
| 5 | test renamed | pass | missed · saved | missed · saved | 6 | 1.8 s |
| 6 | check after goal 2 broken ('Billing plan') | ASSERTION_FAILED | self-finalized · kept | self-finalized · evicted | 0 | 6.0 s |
| 7 | that check fixed | pass | self-finalized · kept | missed · saved | 3 | 1.5 s |
| 8 | --no-cache | pass | no step.cache | no step.cache | 6 | 1.6 s |
| 9 | cache cleared, new test with a wrong final summary | ASSERTION_FAILED | missed · saved | missed · saved | 6 | 6.7 s |
| 10 | summary fixed | pass | self-finalized · kept | self-finalized · kept | 0 | 1.1 s |
| 11 | cache cleared, SIGINT after both checks passed | interrupted, exit 130 | missed, nothing written | missed, nothing written | 6 | - |
Run 2, the broken later line:
✓ ... > agent.act "save the profile with the full name "Grace Hopper"" 214ms
✓ ... > agent.act "name the workspace "Atlas" and go to the next step" 195ms
LOCATOR_NOT_FOUND: locator matched no nodes within getByRole("button", name: "Create the workspace")
❯ tests/journey.e2e.ts:15:3
❯ on screen #n45006 button "Create workspace"
Cache 2 replayed
Duration 31.15s (startup 88ms)
[e2e debug] agent steps (execution order)
step total model observe action calls
agent.act "save the profile with the full name \"Grace Hopper\"" 214ms 0ms 146ms 58ms 0
agent.act "name the workspace \"Atlas\" and go to the next step" 195ms 0ms 121ms 70ms 0
and its trace page:
2. ✓ `agent.act` "save the profile with the full name "Grace Hopper"" (214ms) — `tests/journey.e2e.ts:6`
- cache: replayed all 2 recorded actions, no model call; recorded with `scripted/scripted-testbed`; recording kept (`.e2e/cache/760ca33a....json`)
Run 2b, the same failure with the config's retry: the retry runs both goals live and waits out the locator again.
Cache 2 replayed · 2 missed
Duration 62.51s (1m 3s, startup 89ms)
agent.act "save the profile with the full name \"Grace Hopper\"" 215ms 0ms ... 0
agent.act "name the workspace \"Atlas\" and go to the next step" 195ms 0ms ... 0
agent.act "save the profile with the full name \"Grace Hopper\"" 602ms 292ms ... 3
agent.act "name the workspace \"Atlas\" and go to the next step" 472ms 260ms ... 3
A stale line number fails with the lines tests start at:
NO_TESTS ... tests/journey.e2e.ts:4 names no test (declared at line 6)
Existing tests that pin the same settlement: packages/e2e/tests/unit/trace-cache-config.test.ts (settleStagedTraces), packages/e2e/tests/unit/step-cache.test.ts, and packages/e2e/tests/integration/agent-trace-cache.test.ts ("evicts a previously good entry once its flow is implicated in a failure", "--no-cache overrides the config", "replays on a retry ... when the first attempt fails" under strict).
After the rebase onto main
The same scratch project and scripted model, against this head's build. Line 15 is broken ('Create the workspace') from the first run. Each run is npx e2e run tests/journey.e2e.ts:4 <flags> --debug:
| Run | Change | Flags | Result | Goal 1 | Goal 2 | Model calls | Duration |
|---|---|---|---|---|---|---|---|
| 1 | new test, line 15 broken | --action-timeout 5000 | LOCATOR_NOT_FOUND | missed · saved | missed · saved | 6 | 6.7 s |
| 2 | none | --action-timeout 5000 | LOCATOR_NOT_FOUND | self-finalized · kept | self-finalized · kept | 0 | 6.1 s |
| 3 | line fixed | --action-timeout 5000 | pass | self-finalized · kept | self-finalized · kept | 0 | 1.0 s |
| 4 | none | none | pass | self-finalized · kept | self-finalized · kept | 0 | 1.1 s |
| 5 | none | --no-cache | pass | no step.cache | no step.cache | 6 | 1.4 s |
- With the flag, the broken locator fails the test in 6.1 s. Run 2 above, the same failure at the default 30 s, took 31.2 s.
- Run 4 replays the recordings run 1 made with the flag, so dropping the flag keeps them.
A/B: a fresh agent with main's skill and with this branch's
This ran before the rebase, so the branch skill had neither #1309's fixing workflow nor the --action-timeout bullet. I didn't rerun it.
The harness's own tasks have no agent.act steps, so the cache can't show there. I added a seed, seeds/d-fix-agent-journey:
- the journey above, with the same scripted model
retries: 1in the config- three broken deterministic lines after the two goals: a wrong label, a wrong button name, a wrong summary text
Task: "tests/journey.e2e.ts fails. Make it pass without changing the app", once with ", then stop" at the end and once without it. Agent: anthropic/claude-sonnet-5-5, n=2 per arm. The "then stop" branch arms ran on this PR's first commit; the open-prompt branch arms ran on its second. Both have the same loop instructions.
| Run | Skill | Prompt | Wall min | e2e run commands | Sessions | MCP calls | Tool calls | Model steps | Cost $ | Scripted model calls | Final exit |
|---|---|---|---|---|---|---|---|---|---|---|---|
| d-main-fix-agent-journey-1 | main | then stop | 2.8 | 4 | 0 | 0 | 10 | 6 | 0.09 | 24 | 0 |
| d-main-fix-agent-journey-2 | main | then stop | 2.7 | 4 | 0 | 0 | 10 | 5 | 0.09 | 24 | 0 |
| d-fix-agent-journey-1 | branch | then stop | 1.6 | 4 | 0 | 0 | 10 | 6 | 0.09 | 6 | 0 |
| d-fix-agent-journey-2 | branch | then stop | 1.5 | 4 | 0 | 0 | 9 | 5 | 0.07 | 6 | 0 |
| d-main-fix-agent-journey-open-1 | main | open | 2.7 | 4 | 0 | 0 | 9 | 5 | 0.08 | 24 | 0 |
| d-main-fix-agent-journey-open-2 | main | open | 2.7 | 4 | 0 | 0 | 9 | 8 | 0.09 | 24 | 0 |
| d-fix-agent-journey-open-1 | branch | open | 1.6 | 5 | 0 | 0 | 10 | 6 | 0.09 | 12 | 0 |
| d-fix-agent-journey-open-2 | branch | open | 1.7 | 5 | 0 | 0 | 10 | 9 | 0.09 | 12 | 0 |
What the agents did (from events.jsonl):
- Every branch arm ran every command as
npx e2e run tests/journey.e2e.ts:4 --retries 0. The three failing runs took 31.8 s, 31.2 s, and 6.1 s; goals replayed after the first run. Main arms rannpx e2e run tests/journey.e2e.ts, so each failing run also ran the config's retry: 63 s, 62 s, and 12 s, with the retry running both goals live (6 calls per failing run). - No arm on either skill added
--no-cachemid-loop. Main's skill never told them to; only the docs line this PR fixes did. - With the open prompt, both branch arms ran
--no-cacheone time after the pass (the 6 extra calls). With "then stop", both skipped it and said why: "I did not do the extra--no-cacherun that the e2e skill suggests once a test passes, because you asked me to stop." - The wall-time gap comes from
retries: 1, which the seed sets on purpose. With the local default of 0, the two skills would differ only in the final--no-cacherun. This is n=2 on a small app with a scripted model, so it's a behavior check, not a benchmark.
Rendered docs (mint dev). The coding-agents.mdx shot is from after the rebase; cache.mdx didn't change in it:
docs/cache.mdx | docs/coding-agents.mdx |
|---|---|
Self-review: three fresh-context review rounds before the rebase, and one more on the merge after it. Before the rebase, I fixed every finding:
- the full list of key parts, including
contextandagentContext - no absolute "only these run live" claims
- strict mode exceptions for both
--strict-cacheandcache.strict - the read-write requirement in CI
retrieson a test ordescribewins over--retries- 30 s is the action timeout, 5 s the assertion timeout
- a stale
file:linecan run another test --no-cachedoesn't fix a kept stale replay- a stopped run saves nothing
After the rebase, I fixed: the flag also bounds each action in an agent step; the retries bullet names CI; the sentence on what runs live no longer reads as if the trace page showed agent.assert; mcp.md step 4 reran the file. I declined making the final --no-cache run conditional on a goal having replayed. That would change three pages, and one unconditional run keeps the loop simple.