feat(mcp): run test code in a live session with run_code

Merged
KA
19d ago
+2046-130

A coding agent writing a test over e2e mcp used to walk the flow by node id, write the test from memory, and learn what it got wrong on the first e2e run, where a wrong label waits out the 30 s action timeout. #1317 now prints a checked Test line: for every action. The lines no action prints, an assertion or a step the agent writes without walking it, still meet the app for the first time in e2e run. This adds a run_code catalog tool: the agent sends the statements it's about to write, the session runs them on the live app, and the ones that pass join the test_lines draft.

The tool was first named try_lines; review renamed it run_code, so it no longer reads one letter away from test_lines and says plainly that it runs code against the app. The grammar modules follow it (code-parse.ts, code-surface.ts).

call {tool: "run_code", args: {code: "await screen.getByLabel('Full Name').fill('Ada Lovelace');\nawait screen.getByRole('button', 'Save profile').tap();"}}
Statement 1 of 2 failed, and the one after it did not run. Each statement waited at most 5000 ms.
1. failed: await screen.getByLabel('Full Name').fill('Ada Lovelace');
   LOCATOR_NOT_FOUND: locator matched no nodes within getByLabel("Full Name")
2. not run: await screen.getByRole('button', 'Save profile').tap();

Screen unchanged since revision b6 (re-observed as revision b8, path /forms); the ids you have stay valid.

What changed

  • run_code {code, timeout?} (packages/e2e/src/mcp/run-code.ts) runs statements in order on the session's app and a screen over the session's engine, with matchers from createAsyncExpectation, which is what expect runs for a locator. Waits, the exactly-one rule, and failure messages are a test's. The first failure stops the call and the rest are reported not run. A read (textContent(), count(), ...) shows its value. The result ends with the presenter's screen diff, and a failed statement makes it an MCP error result.
  • No evaluation. code-parse.ts parses the code with oxc-parser, already a runtime dependency (oxcParser() in src/config/typescript.ts), and walks the ESTree nodes into plain data. A statement's source is the code it spans. Every member name is checked against a closed table typed against Screen, Locator, App, and AsyncExpectation (code-surface.ts), in both directions: a name the interface lacks fails to compile, and so does a public member the table neither lists nor leaves out with a reason (Covers<T>), so a member the API gains, like main's Locator.or from #1049, can't drop out of run_code unnoticed. The runner only calls names from that table. Anything else, and code that doesn't parse, is INVALID_ARGUMENT quoting the code with line:column from node offsets, and no statement runs.
  • Limits: 10,000 characters, 50 statements, and 1,000 opening brackets. oxc's parser recurses once per nesting level on the server's own thread, and about 5,300 nested brackets overflow the stack and kill the process (SIGSEGV), every session with it. The deepest code these limits let through, brackets and then arrow functions, parses on a 4 MB stack, half of Node's main-thread default.
  • Timeout: each statement runs under the new AttemptBudget.capped(timeout), which brings the phase deadline forward for the body and puts it back after (default 5000 ms, max 120000). Every locator, matcher, and navigation is already capped by that deadline, so a wrong locator fails at 5 s with the LOCATOR_NOT_FOUND a test gets. The config's own timeouts still apply when shorter.
  • Secrets. Of the two options the brief allowed, this routes through authorization instead of refusing. fill(credentials.user('admin').password) and fill(secrets.get('apiKey')) go through the new LocatorEngine.fillSecret, which reads the node each resolve lands on and runs authorizeSecretFill on it before the value is handed over, again after a stale re-resolve. Same checks and secret.* policy events as type_secret; authorizeSecretFill resolves the value through the attempt's SecretResolver, which raises the pixel taint before the engine types it. The login line is the one an agent most needs to run before writing it, so a refusal would have cost the most there. A Secret anywhere but fill's first argument is a parse error.
  • Guessing a secret through match results is closed two ways.
    • The screen run_code runs on comes from createScreenContext in run/fixtures.ts, the factory the test fixtures use, over a session whose read passes every node through redactNode. A read returns, and a matcher compares, <secret:name> where a value was, so no toContainText, toHaveValue(/…/), or other matcher outcome depends on a secret's characters. A matcher that expects <secret:name> is INVALID_ARGUMENT, since it would pass here and fail in a test.
    • A locator is matched inside the engine against the raw page, attributes and hidden nodes included. So when the config declares any secret, a locator found by part of a text (a regex, an exact other than literal true, hasText) is POLICY_DENIED before anything runs. Exact locators still run.
  • No regex guard. A matcher tests a regex in the server's thread, where timeout can't stop it mid-match. The docs say so, and the skill tells the agent to avoid nested quantifiers.
  • Left out on purpose:
    • Engine fixtures (browser, device). An allowlist of browser.goto and expect(browser).toHaveURL in core would name the web engine's members there (review-triage pattern 13). A statement on one gets a message naming it as an engine fixture, built from the engine.fixtures keys.
    • app.screenshot, which writes an unmasked file after a fill.
    • toHaveScreenshot and expect(screen), which need a test file.
    • setInputFiles, which reads arbitrary paths. upload has the path policy.
    • screen.debug, which prints to the server process.
    • all(), which returns locators.
  • Beside #1317's test lines:
    • The statements that pass, reads aside, join the test_lines draft as written, in the order they ran, each with await and ; (SessionTestLines.ran in packages/e2e/src/mcp/test-lines.ts). Failed and unrun statements add nothing, as a failed action prints no line. Without this, a session that opened a page or filled a form through run_code drafted a body that skipped those steps and started on the wrong screen.
    • run_code observes through the catalog's watched context, so the next action's Test line: is read from the screen run_code returned. Through the unwatched context, an id from a run_code screen on a new page would get No test line: #n.. is not on the newest screen.
    • run_code statements are not rewritten: an app.open with an absolute URL keeps its port, where a navigate line uses the path. Agents write paths here, so I left it.
  • run_code is reserved for project tools beside test_lines (HARNESS_TOOL_NAMES), so MCP telemetry counts it by name through the list #1317 derives from those names. It's named in the server's instructions and the tools description.
  • Docs:
    • reference/mcp.mdx: a run_code entry (inputs, behavior, the grammar table, and one Throws row per refusal), a catalog row, the sessions rule, and test_lines naming the statements run_code passed.
    • mcp-server.mdx: a short "How it tries a test line" section after #1317's "How it writes a test line".
    • reference/security.mdx: two policy-denial rows, and a "where redaction stops" line for locate, see below.
    • tools.mdx and reference/errors.mdx: the reserved name.
  • Skill:
    • mcp.md: a tools row, and workflow step 3: run each line no action printed (an assertion, a step you didn't walk); after a failure, run the fixed statement and the rest from where the passing ones left the app, not from the start.
    • SKILL.md "Workflow: fixing a failed test" step 4 (from #1309) and debugging.md: look live with observe and locate, or run_code for the fixed line and the ones after it.

Verified

Ran it locally: yes, on the web only (no simulator or emulator).

At ed6297d6, after the second review: pnpm check, then the unit tests mcp-code-parse, mcp-test-lines, agent-config, telemetry-events, locator-engine, skill, and the 9 files that use createFixtures (354 tests), and the integration tests mcp-run-code, mcp-test-lines, mcp-server, agent-policy, mcp-session-host, open-session, extended-fixtures, screenshot-assertions, engine-secrets, filled-secrets, screen-tree-secrets, and truncated-secrets (106 tests).

The built server over stdio (packages/e2e/dist/cli/bin.js mcp) on a scratch project: the testbed playground started on port 0, the web engine, and an admin credential. An MCP SDK client script drove it. Node lines are trimmed. The blocks show oxc's syntax error, the bracket bound and the deepest nesting the limits allow refused with the server still answering after them, a statement over several lines with a comment echoed as written, a regex the dropped guard used to refuse now running, and an option fill lacks refused on a secret fill. The last two blocks are a grammar action on an id a run_code screen listed, then the test_lines draft. No output held the credential's value.

Session 01a17ddc-c708-78ac-bfcc-37dcce304d28 open on target "web" (platform web, engine web 0.15.3), headless; config /private/var/folders/97/nd67cfq503j34hr0hdyxtq0m0000gn/T/opencode/ab-authoring/runs/f4-review-smoke/project/e2e.config.ts.
- run_code {code, timeout?}: Run e2e test code against the live session before writing it into a test.

===== passing block: sign in (253 ms) =====
All 5 statements passed.
1. passed: await app.open('/login');
2. passed: await screen.getByLabel('Username').fill(credentials.user('admin').username);
3. passed: await screen.getByLabel('Password').fill(credentials.user('admin').password);
4. passed: await screen.getByRole('button', 'Sign in').tap();
5. passed: await expect(screen.getByRole('status', 'Greeting')).toContainText('admin');

The screen changed substantially since revision b2. Current screen (revision b4, path /dashboard, 28 nodes):
#root document "Dashboard"

===== a regex the removed guard refused, in a matcher (102 ms) =====
The statement passed.
1. passed: await expect(screen.getByRole('status', 'Greeting')).toHaveText(/^Welcome( \w+,?)+!$/);

Screen unchanged since revision b4 (re-observed as revision b6, path /dashboard); the ids you have stay valid.

===== a read, and a chain over several lines with comments, echoed as written (151 ms) =====
All 3 statements passed.
1. passed: await app.open('/forms');
2. passed: await screen
  .getByLabel('Full name') // the name field
  .fill('Grace Hopper');
3. passed: await screen.getByLabel('Full name').inputValue()
   returned "Grace Hopper"

The screen changed substantially since revision b6. Current screen (revision b8, path /forms, 45 nodes):
#root document "Forms"

===== failing locator (5169 ms) [isError] =====
Statement 1 of 2 failed, and the one after it did not run. Each statement waited at most 5000 ms.
1. failed: await screen.getByLabel('Full Name').fill('Ada Lovelace');
   LOCATOR_NOT_FOUND: locator matched no nodes within getByLabel("Full Name")
2. not run: await screen.getByRole('button', 'Save profile').tap();

Screen unchanged since revision b8 (re-observed as revision b10, path /forms); the ids you have stay valid.

===== rejected: outside the grammar (7 ms) [isError] =====
INVALID_ARGUMENT: run_code: "process" at 1:1 is not in the grammar; a statement starts with screen, app, or expect(...)

===== rejected: does not parse (oxc syntax error) (2 ms) [isError] =====
INVALID_ARGUMENT: run_code: the code does not parse at 1:24: Unterminated string

===== rejected: an engine fixture (1 ms) [isError] =====
INVALID_ARGUMENT: run_code: "browser" at 1:14 is a fixture of this target's engine, which run_code does not run; a statement starts with screen, app, or expect(...)

===== rejected: app.screenshot (2 ms) [isError] =====
INVALID_ARGUMENT: run_code: "screenshot" at 1:11 is left out of run_code: it writes an unmasked screenshot file; use the screenshot tool, which withholds pixels after a secret fill

===== rejected: nesting past the bracket bound (2 ms) [isError] =====
INVALID_ARGUMENT: run_code: the code opens 4902 brackets; pass at most 1000 at once

===== rejected: the deepest nesting the limits let through (4 ms) [isError] =====
INVALID_ARGUMENT: run_code: "(((((((((((((((((((((((((((((((((((((((…" at 1:18 is not in the grammar; an argument is a string, number, boolean, null, regular expression, array, or object, a locator on screen, credentials.user('name').username, or a secret as fill's value

===== the server is still up after it (118 ms) =====
The statement passed.
1. passed: await screen.getByLabel('Full name').inputValue();
   returned "Grace Hopper"

Screen unchanged since revision b10 (re-observed as revision b12, path /forms); the ids you have stay valid.

===== partial locator in a project with secrets (6 ms) [isError] =====
POLICY_DENIED: run_code: statement 1 (await screen.getByText(/Grace/).count();) finds a locator by part of a text, and this project configures secrets: the engine matches a regular expression, exact: false, or hasText on unredacted text, so whether it matches could spell a secret out one piece at a time. Find the locator by its exact text; nothing ran.

===== secret into a text field (110 ms) [isError] =====
Statement 1 of 1 failed. Each statement waited at most 5000 ms.
1. failed: await screen.getByLabel('Full name').fill(credentials.user('admin').password);
   POLICY_DENIED: field purpose username is incompatible with secret purpose password

Screen unchanged since revision b12 (re-observed as revision b14, path /forms); the ids you have stay valid.

===== secret fill with an option fill lacks (123 ms) [isError] =====
Statement 1 of 1 failed. Each statement waited at most 5000 ms.
1. failed: await screen.getByLabel('Full name').fill(credentials.user('admin').password, { delay: 5 });
   INVALID_ARGUMENT: fill options has no key "delay"; it takes timeout

Screen unchanged since revision b14 (re-observed as revision b16, path /forms); the ids you have stay valid.

===== screenshot after the password fill (0 images) =====
Screen unchanged since revision b16 (re-observed as revision b18, path /forms); the ids you have stay valid.

No screenshot: a secret was filled in this attempt, so no pixels leave the runner until it ends (PIXEL_TAINTED). Work from the tree and tap listed nodes by id.

===== reading the password field (168 ms) [isError] =====
Statement 3 of 3 failed; the 2 before it passed. Each statement waited at most 5000 ms. The app is where the passing statements left it: run the fixed statement and the rest from here.
1. passed: await app.open('/login');
2. passed: await screen.getByLabel('Password').fill(credentials.user('admin').password);
3. failed: await screen.getByLabel('Password').inputValue();
   POLICY_DENIED: reading values from a secure field is denied: getByLabel("Password")

The screen changed substantially since revision b18. Current screen (revision b20, path /login, 31 nodes):
#root document "Sign in"

===== back on /forms (117 ms) =====
The statement passed.
1. passed: await app.open('/forms');

The screen changed substantially since revision b20. Current screen (revision b22, path /forms, 45 nodes):
#root document "Forms"

===== type into Bio by the id run_code listed (#n60041) =====
Typed into #n60041.
Test line: await screen.getByRole("textbox", "Bio").fill("Wrote compilers");

Screen changes since revision b22 (now revision b24, path /forms, 45 nodes): 1 changed. Every node not listed as removed is still on screen under the id you have.

===== test_lines =====
Draft test body from this session's actions, in order (13 lines):

await app.open();
await app.open('/login');
await screen.getByLabel('Username').fill(credentials.user('admin').username);
await screen.getByLabel('Password').fill(credentials.user('admin').password);
await screen.getByRole('button', 'Sign in').tap();
await expect(screen.getByRole('status', 'Greeting')).toContainText('admin');
await expect(screen.getByRole('status', 'Greeting')).toHaveText(/^Welcome( \w+,?)+!$/);
await app.open('/forms');
await screen
  .getByLabel('Full name') // the name field
  .fill('Grace Hopper');
await app.open('/login');
await screen.getByLabel('Password').fill(credentials.user('admin').password);
await app.open('/forms');
await screen.getByRole("textbox", "Bio").fill("Wrote compilers");

Paste it into a test body, async ({ app, screen }) => { ... }, and add an expect after each step that should change the screen. Import credentials from 'e2e'.

===== secret check =====
21 outputs, 0 contain the credential's password

To check the second redaction pass was redundant, I ran mcp-run-code.test.ts with renderValue's redaction and the server's redacting wrapper both removed: all 11 tests passed, never returns a secret value included. The node-level ledger redacts every read, and the call handler redacts the result with the session's ledger after that.

  • tests/integration/mcp-run-code.test.ts, the built CLI over stdio on the fixture app, covers:
    • a passing block with reads
    • a stop at the first failure inside the timeout
    • ambiguity and assertion messages
    • a refused block running nothing
    • the passing statements in test_lines, reads and failures left out, and the next action's test line read from the screen run_code returned (with run_code observing through the unwatched context, this test fails with No test line: #n42019 is not on the newest screen)
    • secret fills refused for the email field, a button, an unknown credential or secret, an unknown fill option, and a timeout that isn't a number
    • a password field read and toHaveValue, both refused
    • pixels clean after a refused fill and tainted after a real one
    • generic-secret reads redacted
    • matchers comparing redacted text: toContainText('sk-live') fails, toHaveText('<secret:apiKey>') is refused
    • partial locators refused, exact: 0 included
    • the drafted secret fills naming credentials.user('admin').password and secrets.get('apiKey')
    • no returned text containing either secret value
  • tests/unit/mcp-code-parse.test.ts: the grammar, about 75 exact refusal messages (process, constructor, __proto__, import(...), { toString }, assignments, arrow functions, ${} in templates, computed keys, spread, optional chaining, oxc syntax errors), the size and nesting limits with the deepest code they allow, and parsing with eval and Function replaced by throwing stubs.
  • tests/unit/mcp-test-lines.test.ts: run_code statements drafted among the action lines, await and ; added where missing, the credentials import named.
  • tests/unit/locator-engine.test.ts: fillSecret authorizes on every resolve, and capped ends a missing node's wait at the cap and gives the deadline back.

A/B on tasks/author-journey.md (anthropic/claude-sonnet-5-5), run before the rebase and the rename, so the arms called the tool by its old name, try_lines: main's arms had no #1317 test lines, and I didn't re-run it. Arms 1-2 ran on the first commit, which still allowed browser. Arms 3-4 ran on the second, with an observation-based guard since replaced. Arms 5-6 ran on the third commit; the commits after it add a skill line and a refusal (matchers expecting <secret:name>) that none of those arms hit.

RunWall mine2e runsSessionsMCP callsTool callsModel stepsCost $Final exit
main-author-10.8111926170.160
main-author-20.9111723140.140
f4-author-1 (commit 1)0.6111014130.170
f4-author-2 (commit 1)0.6111217150.180
f4-author-3 (commit 2)0.8111018140.210
f4-author-4 (commit 2)0.5111018130.200
f4-author-5 (commit 3)0.6111018130.200
f4-author-6 (commit 3)0.6111018130.200
  • Behavior from events.jsonl: main arms walk the journey one grammar click at a time (navigate 3-4, type 3, type_secret 1, tap 5, select 2, check 1), then write the test. Branch arms replace the clicks with 5-7 run_code calls (then try_lines) of 1-4 statements each, the same screen.* lines they then paste into the test. Every branch arm passed its first e2e run, as both main arms did.
  • No arm hit a refusal. Arms 5-6 used only exact locators, toContainText, and toHaveText.
  • Numbers: MCP calls about halved (17-19 to 10-12). Wall time is within noise here. Cost went up about 30% (0.14-0.16 to 0.20), I think from the longer mcp.md and tool description the agents read. On this small app main's agent doesn't show the slow loop, so this is a behavior check, not a speedup claim. The case run_code is for, a first try that is wrong, doesn't come up in this task.
  • One fix-journey arm on commit 2 (f4-fix-1): 1.5 min, 4 runs, no session, exit 0, the same shape as main-fix-1/2. The agent fixed from terminal output and never opened a session.

Review

Three fresh-context review rounds. Fixed:

  • the regex and substring secret oracle (twice; the observation-based guard of round 1 missed attributes, hidden nodes, and exact: 0)
  • app.screenshot getting past the pixel taint
  • the echoed source rewriting comments and templates
  • toHaveScreenshot that could never run
  • core naming browser members
  • regex catastrophic backtracking (a partial guard, dropped after the second review)
  • run_code missing from the server instructions and the changeset
  • a matcher on <secret:name> passing here and failing in a test
  • the partial-locator check's own query list, now derived from the grammar table so it fails closed

After the rebase, one more fresh-context review of the rebase delta. Fixed: the run_code entry still said to paste passing lines into the test while they also join the draft; the skill's "or a line you changed" would rerun an action #1317 had already printed and leave both lines in the draft; the skill's exact-text advice had no answer for a row, now filter({ has: screen.getByText('Offer B') }).

After the rename and the merge of main, one more fresh-context review. Fixed: run_code refused Locator.or, which the merge brought in from #1049 and writing-tests.md tells agents to use; the tables are now exhaustive against the public interfaces so the next member fails tsc instead. Also renamed the last try-named leftovers (RunCodeResult, the fixture's API key).

After okwasniewski's second review, one more fresh-context review of that delta. Fixed: the bracket bound alone left nesting without brackets (arrow functions, assignments) to the length limit, and at 20,000 characters the deepest code left under a quarter of the stack spare; the limit is now 10,000. And the docs promised every statement stops at timeout, which a backtracking regex doesn't; they say so now.

For the maintainer

  • browser and device: an engine-declared allowlist would let expect(browser).toHaveURL and browser.goto in without core naming them. One shape is a tryable flag on FixtureOperation, read through the FixtureRecorder. It's an engine SPI change, so I left it for a follow-up.
  • Pick one secret policy for partial locators. run_code refuses a regex, an exact other than true, or hasText whenever the config declares a secret, which most projects do (a login credential). The fail-closed reason is that a password shows in plain text behind a "show password" toggle. But locate {text, exact: false} in the same session still answers the substring question; I documented that under "Where redaction stops" and left locate alone. Option (A) applies the same rule to locate, which means changing its zero-match hint and skill step 2, which recommend filter({ hasText }). Option (B) drops the run_code refusal and lists both tools there. I kept the new tool closed and left the existing one to you. Meanwhile the skill tells the agent to try exact text in projects with secrets. #1317 adds a third case: its test lines scope a row with filter({ hasText }), and run_code refuses that line in a project with secrets. The action already ran, so the printed line needs no trying, and the skill points at filter({ has: screen.getByText(...) }) for a row the agent writes itself.
  • Default timeout 5000 ms, which also caps app.open, restart, and clearState (a test gives them config.timeout). A slow first compile can make app.open fail at 5 s. The docs and skill say to raise timeout.