feat(acp): run agent steps on your own coding agent over ACP

Merged
EN
18d ago
+3055-26

Started by @enekesabel; finished by the maintainers on this branch.

What and why

  • New package @e2e-dev/acp: agent.act and agent.assert run on a coding agent the user already signs in to, over the Agent Client Protocol. No model provider or API key for e2e.
  • Presets start the adapter from the project's install (never npx <name>, since unscoped codex-acp on npm is an unrelated package) and turn off the agent's own tools:
    • acpExecutor.claudeCode({ model }): Zed's @agentclientprotocol/claude-agent-acp with tools: [], no settings sources, strict MCP config, no saved session, no permission bypass.
    • acpExecutor.codex(): @agentclientprotocol/codex-acp in read-only mode with shell, web search, apps, plugins, browser, computer use, goals off, and every MCP server from codex mcp list disabled for these sessions.
    • acpExecutor({ command, args }) for any other ACP agent (Cursor's agent acp).
  • One session per test attempt, and one per member of a serial group; later steps are new messages in the same conversation. Tools are the built-in agent's own: createGrammarTools for the target's verbs plus createVerdictTool (complete_step with blocked and its codes), served as one MCP server on 127.0.0.1 (ephemeral port, random path). Every action goes through ctx.actions, so steps record and replay like the built-in agent's, and calls batched behind complete_step are skipped the way the built-in loop skips them.
  • Fail closed (packages/acp/src/session.ts, executor.ts):
    • a permission request is allowed only when the adapter marks it as a step tool call (_meta.claudeCode.toolName, Codex's MCP call marker), or, for other adapters, when its raw input is exactly an MCP call of a step tool; titles never count
    • a tool of the agent's own that runs without asking cancels the turn and fails the step with POLICY_DENIED as soon as it is reported started, and the step tools take no action after it. The session closes with it, so the next step starts a fresh agent that never saw what the tool read
    • the agent's process (and codex mcp list) gets no E2E_SECRET_* / E2E_USER_* variables, in any case, from the runner's env or the executor's own; secrets go by name through type_secret
  • e2e/agent now exports createGrammarTools, createVerdictTool, BASE_RULES, VERDICT_RULES, isRuntimeHardStop, and the ScreenOutput type (.changeset/grammar-tools-export.md, asserted in tests/types/sdk-types.ts, including that the screen presenter and onActionFailed stay internal). @e2e-dev/acp peers on e2e >=0.21.4 <1, the release that ships them.
import { acpExecutor } from '@e2e-dev/acp';

export default {
  targets: [{ engine: web(), app: { url: 'http://localhost:3000' } }],
  agents: { default: { executor: acpExecutor.claudeCode({ model: 'sonnet' }) } },
} satisfies E2EConfig;
sequenceDiagram
  participant R as e2e runner
  participant X as acpExecutor
  participant A as ACP agent (stdio)
  participant T as step tools (MCP on 127.0.0.1)
  R->>X: runStep(ctx)
  X->>A: initialize, session/new { mcpServers: [T] } (first step only)
  X->>A: session/prompt (step, params, ledger, screen)
  A->>X: request_permission (step tool: allow, anything else: reject)
  A->>T: tap { target: "n7" }
  T->>R: ctx.actions.tap
  T-->>A: "Tapped #n7." + screen changes
  A->>X: tool_call of its own started without asking: cancel, POLICY_DENIED, close session
  A->>T: complete_step { status, summary, errorCode? }
  A-->>X: end_turn (usage, cost)
  X-->>R: verdict, recordModelCall

Known limits

  • Secrets the config reads from other variables (secrets: { apiKey: process.env.API_KEY }), or the variable an E2E_SECRET_* value was copied from, still reach the agent's process. The executor sees secret names only, so stripping by value would need a core API; docs/acp.mdx says to keep secrets in the E2E_ variables.
  • The Claude adapter passes the step-tool URL to claude as --mcp-config on its command line, so another user on the same machine can read it with ps and reach the step tools while a test runs. The MCP handler is stateless and a header token would sit on the same command line.
  • After complete_step the turn runs to its end, so the agent writes its closing reply (one more model round trip) before the verdict returns. Cancelling the turn instead would drop the usage and cost the adapter reports when the turn ends.
  • A turn that times out records no model call, and an agent slow to honor a cancel fails the next step instead (did not stop the turn of a step that already ended).
  • On Windows, a custom command must be an executable, not an npm .cmd shim (documented).
  • No Cursor preset: Cursor isn't installed here, and its MCP permission requests may not carry the fields the check requires, so it ships as acpExecutor({ command: 'agent', args: ['acp'] }) until someone runs it.
  • Codex can still start subagents (agents.max_threads must be at least 1); a step where it does fails with POLICY_DENIED.
  • Adapters resolve from the working directory up; e2e run --config apps/web/e2e.config.ts from a monorepo root needs the adapter installed where e2e runs (the runner does not pass the config directory to executors).
  • @e2e-dev/acp is a new npm package: publish its first version by hand and connect trusted publishing before this changeset releases.

Verified

Ran it locally: yes

  • packages/acp/tests/cli.test.ts: built CLI on real Chromium with a scripted ACP agent process; act + assert pass, type_secret fills by name and the value appears in no prompt, tool result, or artifact, and a second run replays both acts from the cache.
  • packages/acp/tests/executor.test.ts, presets.test.ts (38 tests). Each test for the behaviors below fails on the code before its fix: a serial group's second member gets its own session, a tap batched with complete_step is skipped, an own tool fails the step when it starts, an allowed step-tool call is not flagged as the agent's own once it completes, no prompt goes out for a step that ended while its screen was read, the session closes after an own tool (also one run in the tail of an ended step), and secret variables are stripped in any case and from codex mcp list.
  • Live, node packages/acp/tests/live/acp.ts on Linux against the built packages at this head (record, then replay): a todo act + assert, an assertion that must fail, a secret login, and a step that asks the agent to list its tools.
AgentRecordReplayAgent's own tool callsSecret in artifacts
Claude Code (claude-agent-acp 0.97.0, sonnet)4/4 as expected, wrong assertion fails ASSERTION_FAILEDacts from cache, 0 model calls, 0.2 to 0.3 snoneno
Codex (codex-acp 2.1.1, gpt-6.1-sol), macOS, before the takeover; not re-run at this head, the local OpenAI account is over quota4/4 as expected, steps 6 to 13 sacts from cache, 0 model calls, 0.2 snoneno
  • Serial group, live on Claude Code: a scratch project with test.describe('todo flow', { serial: true }) whose second test adds a todo on the page the first left.
Before the per-member sessionThis branch
second test fails: MODEL_PROVIDER_FAILED: The ACP agent session failed: ACP connection closedboth pass live (1 model call each, $0.03), then replay from the cache in 1.05 s
  • Containment, asking each preset's agent to list and then try its own tools (macOS run before the takeover): Claude Code lists only mcp__e2e_step__*. Codex has no shell (exec_command is not a function), no web search, and its code-mode exec cannot require; a subagent spawn trips the own-tool check.

Takeover changes made with Claude Opus 5.5 in Claude Code.