feat(mcp): run test code in a live session with run_code
A coding agent writing a test over e2e mcp used to walk the flow by node id, write the test from memory, and learn what it got wrong on the first e2e run, where a wrong label waits out the 30 s action timeout. #1317 now prints a checked Test line: for every action. The lines no action prints, an assertion or a step the agent writes without walking it, still meet the app for the first time in e2e run. This adds a run_code catalog tool: the agent sends the statements it's about to write, the session runs them on the live app, and the ones that pass join the test_lines draft.
The tool was first named try_lines; review renamed it run_code, so it no longer reads one letter away from test_lines and says plainly that it runs code against the app. The grammar modules follow it (code-parse.ts, code-surface.ts).
call {tool: "run_code", args: {code: "await screen.getByLabel('Full Name').fill('Ada Lovelace');\nawait screen.getByRole('button', 'Save profile').tap();"}}
Statement 1 of 2 failed, and the one after it did not run. Each statement waited at most 5000 ms.
1. failed: await screen.getByLabel('Full Name').fill('Ada Lovelace');
LOCATOR_NOT_FOUND: locator matched no nodes within getByLabel("Full Name")
2. not run: await screen.getByRole('button', 'Save profile').tap();
Screen unchanged since revision b6 (re-observed as revision b8, path /forms); the ids you have stay valid.
What changed
run_code {code, timeout?}(packages/e2e/src/mcp/run-code.ts) runs statements in order on the session'sappand ascreenover the session's engine, with matchers fromcreateAsyncExpectation, which is whatexpectruns for a locator. Waits, the exactly-one rule, and failure messages are a test's. The first failure stops the call and the rest are reportednot run. A read (textContent(),count(), ...) shows its value. The result ends with the presenter's screen diff, and a failed statement makes it an MCP error result.- No evaluation.
code-parse.tsparses the code with oxc-parser, already a runtime dependency (oxcParser()insrc/config/typescript.ts), and walks the ESTree nodes into plain data. A statement's source is the code it spans. Every member name is checked against a closed table typed againstScreen,Locator,App, andAsyncExpectation(code-surface.ts), in both directions: a name the interface lacks fails to compile, and so does a public member the table neither lists nor leaves out with a reason (Covers<T>), so a member the API gains, like main'sLocator.orfrom #1049, can't drop out ofrun_codeunnoticed. The runner only calls names from that table. Anything else, and code that doesn't parse, isINVALID_ARGUMENTquoting the code withline:columnfrom node offsets, and no statement runs. - Limits: 10,000 characters, 50 statements, and 1,000 opening brackets. oxc's parser recurses once per nesting level on the server's own thread, and about 5,300 nested brackets overflow the stack and kill the process (SIGSEGV), every session with it. The deepest code these limits let through, brackets and then arrow functions, parses on a 4 MB stack, half of Node's main-thread default.
- Timeout: each statement runs under the new
AttemptBudget.capped(timeout), which brings the phase deadline forward for the body and puts it back after (default 5000 ms, max 120000). Every locator, matcher, and navigation is already capped by that deadline, so a wrong locator fails at 5 s with theLOCATOR_NOT_FOUNDa test gets. The config's own timeouts still apply when shorter. - Secrets. Of the two options the brief allowed, this routes through authorization instead of refusing.
fill(credentials.user('admin').password)andfill(secrets.get('apiKey'))go through the newLocatorEngine.fillSecret, which reads the node each resolve lands on and runsauthorizeSecretFillon it before the value is handed over, again after a stale re-resolve. Same checks andsecret.*policy events astype_secret;authorizeSecretFillresolves the value through the attempt'sSecretResolver, which raises the pixel taint before the engine types it. The login line is the one an agent most needs to run before writing it, so a refusal would have cost the most there. A Secret anywhere butfill's first argument is a parse error. - Guessing a secret through match results is closed two ways.
- The
screenrun_code runs on comes fromcreateScreenContextinrun/fixtures.ts, the factory the test fixtures use, over a session whosereadpasses every node throughredactNode. A read returns, and a matcher compares,<secret:name>where a value was, so notoContainText,toHaveValue(/…/), or other matcher outcome depends on a secret's characters. A matcher that expects<secret:name>isINVALID_ARGUMENT, since it would pass here and fail in a test. - A locator is matched inside the engine against the raw page, attributes and hidden nodes included. So when the config declares any secret, a locator found by part of a text (a regex, an
exactother than literaltrue,hasText) isPOLICY_DENIEDbefore anything runs. Exact locators still run.
- The
- No regex guard. A matcher tests a regex in the server's thread, where
timeoutcan't stop it mid-match. The docs say so, and the skill tells the agent to avoid nested quantifiers. - Left out on purpose:
- Engine fixtures (
browser,device). An allowlist ofbrowser.gotoandexpect(browser).toHaveURLin core would name the web engine's members there (review-triage pattern 13). A statement on one gets a message naming it as an engine fixture, built from theengine.fixtureskeys. app.screenshot, which writes an unmasked file after a fill.toHaveScreenshotandexpect(screen), which need a test file.setInputFiles, which reads arbitrary paths.uploadhas the path policy.screen.debug, which prints to the server process.all(), which returns locators.
- Engine fixtures (
- Beside #1317's test lines:
- The statements that pass, reads aside, join the
test_linesdraft as written, in the order they ran, each withawaitand;(SessionTestLines.raninpackages/e2e/src/mcp/test-lines.ts). Failed and unrun statements add nothing, as a failed action prints no line. Without this, a session that opened a page or filled a form throughrun_codedrafted a body that skipped those steps and started on the wrong screen. run_codeobserves through the catalog's watched context, so the next action'sTest line:is read from the screenrun_codereturned. Through the unwatched context, an id from arun_codescreen on a new page would getNo test line: #n.. is not on the newest screen.run_codestatements are not rewritten: anapp.openwith an absolute URL keeps its port, where anavigateline uses the path. Agents write paths here, so I left it.
- The statements that pass, reads aside, join the
run_codeis reserved for project tools besidetest_lines(HARNESS_TOOL_NAMES), so MCP telemetry counts it by name through the list #1317 derives from those names. It's named in the server's instructions and thetoolsdescription.- Docs:
reference/mcp.mdx: arun_codeentry (inputs, behavior, the grammar table, and one Throws row per refusal), a catalog row, the sessions rule, andtest_linesnaming the statementsrun_codepassed.mcp-server.mdx: a short "How it tries a test line" section after #1317's "How it writes a test line".reference/security.mdx: two policy-denial rows, and a "where redaction stops" line forlocate, see below.tools.mdxandreference/errors.mdx: the reserved name.
- Skill:
mcp.md: a tools row, and workflow step 3: run each line no action printed (an assertion, a step you didn't walk); after a failure, run the fixed statement and the rest from where the passing ones left the app, not from the start.SKILL.md"Workflow: fixing a failed test" step 4 (from #1309) anddebugging.md: look live withobserveandlocate, orrun_codefor the fixed line and the ones after it.
Verified
Ran it locally: yes, on the web only (no simulator or emulator).
At ed6297d6, after the second review: pnpm check, then the unit tests mcp-code-parse, mcp-test-lines, agent-config, telemetry-events, locator-engine, skill, and the 9 files that use createFixtures (354 tests), and the integration tests mcp-run-code, mcp-test-lines, mcp-server, agent-policy, mcp-session-host, open-session, extended-fixtures, screenshot-assertions, engine-secrets, filled-secrets, screen-tree-secrets, and truncated-secrets (106 tests).
The built server over stdio (packages/e2e/dist/cli/bin.js mcp) on a scratch project: the testbed playground started on port 0, the web engine, and an admin credential. An MCP SDK client script drove it. Node lines are trimmed. The blocks show oxc's syntax error, the bracket bound and the deepest nesting the limits allow refused with the server still answering after them, a statement over several lines with a comment echoed as written, a regex the dropped guard used to refuse now running, and an option fill lacks refused on a secret fill. The last two blocks are a grammar action on an id a run_code screen listed, then the test_lines draft. No output held the credential's value.
Session 01a17ddc-c708-78ac-bfcc-37dcce304d28 open on target "web" (platform web, engine web 0.15.3), headless; config /private/var/folders/97/nd67cfq503j34hr0hdyxtq0m0000gn/T/opencode/ab-authoring/runs/f4-review-smoke/project/e2e.config.ts.
- run_code {code, timeout?}: Run e2e test code against the live session before writing it into a test.
===== passing block: sign in (253 ms) =====
All 5 statements passed.
1. passed: await app.open('/login');
2. passed: await screen.getByLabel('Username').fill(credentials.user('admin').username);
3. passed: await screen.getByLabel('Password').fill(credentials.user('admin').password);
4. passed: await screen.getByRole('button', 'Sign in').tap();
5. passed: await expect(screen.getByRole('status', 'Greeting')).toContainText('admin');
The screen changed substantially since revision b2. Current screen (revision b4, path /dashboard, 28 nodes):
#root document "Dashboard"
===== a regex the removed guard refused, in a matcher (102 ms) =====
The statement passed.
1. passed: await expect(screen.getByRole('status', 'Greeting')).toHaveText(/^Welcome( \w+,?)+!$/);
Screen unchanged since revision b4 (re-observed as revision b6, path /dashboard); the ids you have stay valid.
===== a read, and a chain over several lines with comments, echoed as written (151 ms) =====
All 3 statements passed.
1. passed: await app.open('/forms');
2. passed: await screen
.getByLabel('Full name') // the name field
.fill('Grace Hopper');
3. passed: await screen.getByLabel('Full name').inputValue()
returned "Grace Hopper"
The screen changed substantially since revision b6. Current screen (revision b8, path /forms, 45 nodes):
#root document "Forms"
===== failing locator (5169 ms) [isError] =====
Statement 1 of 2 failed, and the one after it did not run. Each statement waited at most 5000 ms.
1. failed: await screen.getByLabel('Full Name').fill('Ada Lovelace');
LOCATOR_NOT_FOUND: locator matched no nodes within getByLabel("Full Name")
2. not run: await screen.getByRole('button', 'Save profile').tap();
Screen unchanged since revision b8 (re-observed as revision b10, path /forms); the ids you have stay valid.
===== rejected: outside the grammar (7 ms) [isError] =====
INVALID_ARGUMENT: run_code: "process" at 1:1 is not in the grammar; a statement starts with screen, app, or expect(...)
===== rejected: does not parse (oxc syntax error) (2 ms) [isError] =====
INVALID_ARGUMENT: run_code: the code does not parse at 1:24: Unterminated string
===== rejected: an engine fixture (1 ms) [isError] =====
INVALID_ARGUMENT: run_code: "browser" at 1:14 is a fixture of this target's engine, which run_code does not run; a statement starts with screen, app, or expect(...)
===== rejected: app.screenshot (2 ms) [isError] =====
INVALID_ARGUMENT: run_code: "screenshot" at 1:11 is left out of run_code: it writes an unmasked screenshot file; use the screenshot tool, which withholds pixels after a secret fill
===== rejected: nesting past the bracket bound (2 ms) [isError] =====
INVALID_ARGUMENT: run_code: the code opens 4902 brackets; pass at most 1000 at once
===== rejected: the deepest nesting the limits let through (4 ms) [isError] =====
INVALID_ARGUMENT: run_code: "(((((((((((((((((((((((((((((((((((((((…" at 1:18 is not in the grammar; an argument is a string, number, boolean, null, regular expression, array, or object, a locator on screen, credentials.user('name').username, or a secret as fill's value
===== the server is still up after it (118 ms) =====
The statement passed.
1. passed: await screen.getByLabel('Full name').inputValue();
returned "Grace Hopper"
Screen unchanged since revision b10 (re-observed as revision b12, path /forms); the ids you have stay valid.
===== partial locator in a project with secrets (6 ms) [isError] =====
POLICY_DENIED: run_code: statement 1 (await screen.getByText(/Grace/).count();) finds a locator by part of a text, and this project configures secrets: the engine matches a regular expression, exact: false, or hasText on unredacted text, so whether it matches could spell a secret out one piece at a time. Find the locator by its exact text; nothing ran.
===== secret into a text field (110 ms) [isError] =====
Statement 1 of 1 failed. Each statement waited at most 5000 ms.
1. failed: await screen.getByLabel('Full name').fill(credentials.user('admin').password);
POLICY_DENIED: field purpose username is incompatible with secret purpose password
Screen unchanged since revision b12 (re-observed as revision b14, path /forms); the ids you have stay valid.
===== secret fill with an option fill lacks (123 ms) [isError] =====
Statement 1 of 1 failed. Each statement waited at most 5000 ms.
1. failed: await screen.getByLabel('Full name').fill(credentials.user('admin').password, { delay: 5 });
INVALID_ARGUMENT: fill options has no key "delay"; it takes timeout
Screen unchanged since revision b14 (re-observed as revision b16, path /forms); the ids you have stay valid.
===== screenshot after the password fill (0 images) =====
Screen unchanged since revision b16 (re-observed as revision b18, path /forms); the ids you have stay valid.
No screenshot: a secret was filled in this attempt, so no pixels leave the runner until it ends (PIXEL_TAINTED). Work from the tree and tap listed nodes by id.
===== reading the password field (168 ms) [isError] =====
Statement 3 of 3 failed; the 2 before it passed. Each statement waited at most 5000 ms. The app is where the passing statements left it: run the fixed statement and the rest from here.
1. passed: await app.open('/login');
2. passed: await screen.getByLabel('Password').fill(credentials.user('admin').password);
3. failed: await screen.getByLabel('Password').inputValue();
POLICY_DENIED: reading values from a secure field is denied: getByLabel("Password")
The screen changed substantially since revision b18. Current screen (revision b20, path /login, 31 nodes):
#root document "Sign in"
===== back on /forms (117 ms) =====
The statement passed.
1. passed: await app.open('/forms');
The screen changed substantially since revision b20. Current screen (revision b22, path /forms, 45 nodes):
#root document "Forms"
===== type into Bio by the id run_code listed (#n60041) =====
Typed into #n60041.
Test line: await screen.getByRole("textbox", "Bio").fill("Wrote compilers");
Screen changes since revision b22 (now revision b24, path /forms, 45 nodes): 1 changed. Every node not listed as removed is still on screen under the id you have.
===== test_lines =====
Draft test body from this session's actions, in order (13 lines):
await app.open();
await app.open('/login');
await screen.getByLabel('Username').fill(credentials.user('admin').username);
await screen.getByLabel('Password').fill(credentials.user('admin').password);
await screen.getByRole('button', 'Sign in').tap();
await expect(screen.getByRole('status', 'Greeting')).toContainText('admin');
await expect(screen.getByRole('status', 'Greeting')).toHaveText(/^Welcome( \w+,?)+!$/);
await app.open('/forms');
await screen
.getByLabel('Full name') // the name field
.fill('Grace Hopper');
await app.open('/login');
await screen.getByLabel('Password').fill(credentials.user('admin').password);
await app.open('/forms');
await screen.getByRole("textbox", "Bio").fill("Wrote compilers");
Paste it into a test body, async ({ app, screen }) => { ... }, and add an expect after each step that should change the screen. Import credentials from 'e2e'.
===== secret check =====
21 outputs, 0 contain the credential's password
To check the second redaction pass was redundant, I ran mcp-run-code.test.ts with renderValue's redaction and the server's redacting wrapper both removed: all 11 tests passed, never returns a secret value included. The node-level ledger redacts every read, and the call handler redacts the result with the session's ledger after that.
tests/integration/mcp-run-code.test.ts, the built CLI over stdio on the fixture app, covers:- a passing block with reads
- a stop at the first failure inside the timeout
- ambiguity and assertion messages
- a refused block running nothing
- the passing statements in
test_lines, reads and failures left out, and the next action's test line read from the screenrun_codereturned (withrun_codeobserving through the unwatched context, this test fails withNo test line: #n42019 is not on the newest screen) - secret fills refused for the email field, a button, an unknown credential or secret, an unknown
filloption, and a timeout that isn't a number - a password field read and
toHaveValue, both refused - pixels clean after a refused fill and tainted after a real one
- generic-secret reads redacted
- matchers comparing redacted text:
toContainText('sk-live')fails,toHaveText('<secret:apiKey>')is refused - partial locators refused,
exact: 0included - the drafted secret fills naming
credentials.user('admin').passwordandsecrets.get('apiKey') - no returned text containing either secret value
tests/unit/mcp-code-parse.test.ts: the grammar, about 75 exact refusal messages (process,constructor,__proto__,import(...),{ toString }, assignments, arrow functions,${}in templates, computed keys, spread, optional chaining, oxc syntax errors), the size and nesting limits with the deepest code they allow, and parsing withevalandFunctionreplaced by throwing stubs.tests/unit/mcp-test-lines.test.ts:run_codestatements drafted among the action lines,awaitand;added where missing, thecredentialsimport named.tests/unit/locator-engine.test.ts:fillSecretauthorizes on every resolve, andcappedends a missing node's wait at the cap and gives the deadline back.
A/B on tasks/author-journey.md (anthropic/claude-sonnet-5-5), run before the rebase and the rename, so the arms called the tool by its old name, try_lines: main's arms had no #1317 test lines, and I didn't re-run it. Arms 1-2 ran on the first commit, which still allowed browser. Arms 3-4 ran on the second, with an observation-based guard since replaced. Arms 5-6 ran on the third commit; the commits after it add a skill line and a refusal (matchers expecting <secret:name>) that none of those arms hit.
| Run | Wall min | e2e runs | Sessions | MCP calls | Tool calls | Model steps | Cost $ | Final exit |
|---|---|---|---|---|---|---|---|---|
| main-author-1 | 0.8 | 1 | 1 | 19 | 26 | 17 | 0.16 | 0 |
| main-author-2 | 0.9 | 1 | 1 | 17 | 23 | 14 | 0.14 | 0 |
| f4-author-1 (commit 1) | 0.6 | 1 | 1 | 10 | 14 | 13 | 0.17 | 0 |
| f4-author-2 (commit 1) | 0.6 | 1 | 1 | 12 | 17 | 15 | 0.18 | 0 |
| f4-author-3 (commit 2) | 0.8 | 1 | 1 | 10 | 18 | 14 | 0.21 | 0 |
| f4-author-4 (commit 2) | 0.5 | 1 | 1 | 10 | 18 | 13 | 0.20 | 0 |
| f4-author-5 (commit 3) | 0.6 | 1 | 1 | 10 | 18 | 13 | 0.20 | 0 |
| f4-author-6 (commit 3) | 0.6 | 1 | 1 | 10 | 18 | 13 | 0.20 | 0 |
- Behavior from
events.jsonl: main arms walk the journey one grammar click at a time (navigate3-4,type3,type_secret1,tap5,select2,check1), then write the test. Branch arms replace the clicks with 5-7run_codecalls (thentry_lines) of 1-4 statements each, the samescreen.*lines they then paste into the test. Every branch arm passed its firste2e run, as both main arms did. - No arm hit a refusal. Arms 5-6 used only exact locators,
toContainText, andtoHaveText. - Numbers: MCP calls about halved (17-19 to 10-12). Wall time is within noise here. Cost went up about 30% (0.14-0.16 to 0.20), I think from the longer
mcp.mdand tool description the agents read. On this small app main's agent doesn't show the slow loop, so this is a behavior check, not a speedup claim. The caserun_codeis for, a first try that is wrong, doesn't come up in this task. - One
fix-journeyarm on commit 2 (f4-fix-1): 1.5 min, 4 runs, no session, exit 0, the same shape asmain-fix-1/2. The agent fixed from terminal output and never opened a session.
Review
Three fresh-context review rounds. Fixed:
- the regex and substring secret oracle (twice; the observation-based guard of round 1 missed attributes, hidden nodes, and
exact: 0) app.screenshotgetting past the pixel taint- the echoed source rewriting comments and templates
toHaveScreenshotthat could never run- core naming
browsermembers - regex catastrophic backtracking (a partial guard, dropped after the second review)
run_codemissing from the server instructions and the changeset- a matcher on
<secret:name>passing here and failing in a test - the partial-locator check's own query list, now derived from the grammar table so it fails closed
After the rebase, one more fresh-context review of the rebase delta. Fixed: the run_code entry still said to paste passing lines into the test while they also join the draft; the skill's "or a line you changed" would rerun an action #1317 had already printed and leave both lines in the draft; the skill's exact-text advice had no answer for a row, now filter({ has: screen.getByText('Offer B') }).
After the rename and the merge of main, one more fresh-context review. Fixed: run_code refused Locator.or, which the merge brought in from #1049 and writing-tests.md tells agents to use; the tables are now exhaustive against the public interfaces so the next member fails tsc instead. Also renamed the last try-named leftovers (RunCodeResult, the fixture's API key).
After okwasniewski's second review, one more fresh-context review of that delta. Fixed: the bracket bound alone left nesting without brackets (arrow functions, assignments) to the length limit, and at 20,000 characters the deepest code left under a quarter of the stack spare; the limit is now 10,000. And the docs promised every statement stops at timeout, which a backtracking regex doesn't; they say so now.
For the maintainer
browseranddevice: an engine-declared allowlist would letexpect(browser).toHaveURLandbrowser.gotoin without core naming them. One shape is atryableflag onFixtureOperation, read through theFixtureRecorder. It's an engine SPI change, so I left it for a follow-up.- Pick one secret policy for partial locators.
run_coderefuses a regex, anexactother thantrue, orhasTextwhenever the config declares a secret, which most projects do (a login credential). The fail-closed reason is that a password shows in plain text behind a "show password" toggle. Butlocate {text, exact: false}in the same session still answers the substring question; I documented that under "Where redaction stops" and leftlocatealone. Option (A) applies the same rule tolocate, which means changing its zero-match hint and skill step 2, which recommendfilter({ hasText }). Option (B) drops therun_coderefusal and lists both tools there. I kept the new tool closed and left the existing one to you. Meanwhile the skill tells the agent to try exact text in projects with secrets. #1317 adds a third case: its test lines scope a row withfilter({ hasText }), andrun_coderefuses that line in a project with secrets. The action already ran, so the printed line needs no trying, and the skill points atfilter({ has: screen.getByText(...) })for a row the agent writes itself. - Default timeout 5000 ms, which also caps
app.open,restart, andclearState(a test gives themconfig.timeout). A slow first compile can makeapp.openfail at 5 s. The docs and skill say to raisetimeout.