feat(acp): run agent steps on your own coding agent over ACP
Merged
EN
18d ago+3055+3055+0123456789012345678901234567890123456789-26-26-01234567890123456789
Started by @enekesabel; finished by the maintainers on this branch.
What and why
- New package
@e2e-dev/acp:agent.actandagent.assertrun on a coding agent the user already signs in to, over the Agent Client Protocol. No model provider or API key for e2e. - Presets start the adapter from the project's install (never
npx <name>, since unscopedcodex-acpon npm is an unrelated package) and turn off the agent's own tools:acpExecutor.claudeCode({ model }): Zed's@agentclientprotocol/claude-agent-acpwithtools: [], no settings sources, strict MCP config, no saved session, no permission bypass.acpExecutor.codex():@agentclientprotocol/codex-acpin read-only mode with shell, web search, apps, plugins, browser, computer use, goals off, and every MCP server fromcodex mcp listdisabled for these sessions.acpExecutor({ command, args })for any other ACP agent (Cursor'sagent acp).
- One session per test attempt, and one per member of a serial group; later steps are new messages in the same conversation. Tools are the built-in agent's own:
createGrammarToolsfor the target's verbs pluscreateVerdictTool(complete_stepwithblockedand its codes), served as one MCP server on 127.0.0.1 (ephemeral port, random path). Every action goes throughctx.actions, so steps record and replay like the built-in agent's, and calls batched behindcomplete_stepare skipped the way the built-in loop skips them. - Fail closed (
packages/acp/src/session.ts,executor.ts):- a permission request is allowed only when the adapter marks it as a step tool call (
_meta.claudeCode.toolName, Codex's MCP call marker), or, for other adapters, when its raw input is exactly an MCP call of a step tool; titles never count - a tool of the agent's own that runs without asking cancels the turn and fails the step with
POLICY_DENIEDas soon as it is reported started, and the step tools take no action after it. The session closes with it, so the next step starts a fresh agent that never saw what the tool read - the agent's process (and
codex mcp list) gets noE2E_SECRET_*/E2E_USER_*variables, in any case, from the runner's env or the executor's own; secrets go by name throughtype_secret
- a permission request is allowed only when the adapter marks it as a step tool call (
e2e/agentnow exportscreateGrammarTools,createVerdictTool,BASE_RULES,VERDICT_RULES,isRuntimeHardStop, and theScreenOutputtype (.changeset/grammar-tools-export.md, asserted intests/types/sdk-types.ts, including that the screen presenter andonActionFailedstay internal).@e2e-dev/acppeers one2e >=0.21.4 <1, the release that ships them.
import { acpExecutor } from '@e2e-dev/acp';
export default {
targets: [{ engine: web(), app: { url: 'http://localhost:3000' } }],
agents: { default: { executor: acpExecutor.claudeCode({ model: 'sonnet' }) } },
} satisfies E2EConfig;
sequenceDiagram
participant R as e2e runner
participant X as acpExecutor
participant A as ACP agent (stdio)
participant T as step tools (MCP on 127.0.0.1)
R->>X: runStep(ctx)
X->>A: initialize, session/new { mcpServers: [T] } (first step only)
X->>A: session/prompt (step, params, ledger, screen)
A->>X: request_permission (step tool: allow, anything else: reject)
A->>T: tap { target: "n7" }
T->>R: ctx.actions.tap
T-->>A: "Tapped #n7." + screen changes
A->>X: tool_call of its own started without asking: cancel, POLICY_DENIED, close session
A->>T: complete_step { status, summary, errorCode? }
A-->>X: end_turn (usage, cost)
X-->>R: verdict, recordModelCall
Known limits
- Secrets the config reads from other variables (
secrets: { apiKey: process.env.API_KEY }), or the variable anE2E_SECRET_*value was copied from, still reach the agent's process. The executor sees secret names only, so stripping by value would need a core API;docs/acp.mdxsays to keep secrets in theE2E_variables. - The Claude adapter passes the step-tool URL to
claudeas--mcp-configon its command line, so another user on the same machine can read it withpsand reach the step tools while a test runs. The MCP handler is stateless and a header token would sit on the same command line. - After
complete_stepthe turn runs to its end, so the agent writes its closing reply (one more model round trip) before the verdict returns. Cancelling the turn instead would drop the usage and cost the adapter reports when the turn ends. - A turn that times out records no model call, and an agent slow to honor a cancel fails the next step instead (
did not stop the turn of a step that already ended). - On Windows, a custom
commandmust be an executable, not an npm.cmdshim (documented). - No Cursor preset: Cursor isn't installed here, and its MCP permission requests may not carry the fields the check requires, so it ships as
acpExecutor({ command: 'agent', args: ['acp'] })until someone runs it. - Codex can still start subagents (
agents.max_threadsmust be at least 1); a step where it does fails withPOLICY_DENIED. - Adapters resolve from the working directory up;
e2e run --config apps/web/e2e.config.tsfrom a monorepo root needs the adapter installed where e2e runs (the runner does not pass the config directory to executors). @e2e-dev/acpis a new npm package: publish its first version by hand and connect trusted publishing before this changeset releases.
Verified
Ran it locally: yes
packages/acp/tests/cli.test.ts: built CLI on real Chromium with a scripted ACP agent process; act + assert pass,type_secretfills by name and the value appears in no prompt, tool result, or artifact, and a second run replays both acts from the cache.packages/acp/tests/executor.test.ts,presets.test.ts(38 tests). Each test for the behaviors below fails on the code before its fix: a serial group's second member gets its own session, atapbatched withcomplete_stepis skipped, an own tool fails the step when it starts, an allowed step-tool call is not flagged as the agent's own once it completes, no prompt goes out for a step that ended while its screen was read, the session closes after an own tool (also one run in the tail of an ended step), and secret variables are stripped in any case and fromcodex mcp list.- Live,
node packages/acp/tests/live/acp.tson Linux against the built packages at this head (record, then replay): a todo act + assert, an assertion that must fail, a secret login, and a step that asks the agent to list its tools.
| Agent | Record | Replay | Agent's own tool calls | Secret in artifacts |
|---|---|---|---|---|
Claude Code (claude-agent-acp 0.97.0, sonnet) | 4/4 as expected, wrong assertion fails ASSERTION_FAILED | acts from cache, 0 model calls, 0.2 to 0.3 s | none | no |
Codex (codex-acp 2.1.1, gpt-6.1-sol), macOS, before the takeover; not re-run at this head, the local OpenAI account is over quota | 4/4 as expected, steps 6 to 13 s | acts from cache, 0 model calls, 0.2 s | none | no |
- Serial group, live on Claude Code: a scratch project with
test.describe('todo flow', { serial: true })whose second test adds a todo on the page the first left.
| Before the per-member session | This branch |
|---|---|
second test fails: MODEL_PROVIDER_FAILED: The ACP agent session failed: ACP connection closed | both pass live (1 model call each, $0.03), then replay from the cache in 1.05 s |
- Containment, asking each preset's agent to list and then try its own tools (macOS run before the takeover): Claude Code lists only
mcp__e2e_step__*. Codex has no shell (exec_commandis not a function), no web search, and its code-modeexeccannotrequire; a subagent spawn trips the own-tool check.
Takeover changes made with Claude Opus 5.5 in Claude Code.