feat(runner): expose explore() through e2e/runner

Merged
AR
21d ago
+569-39

Closes #1196.

e2e/runner gets explore(), the counterpart of e2e explore that run() is to e2e run.

import type { E2EConfig } from 'e2e';
import { explore } from 'e2e/runner';

const result = await explore({
  cwd: projectDirectory,
  config, // an E2EConfig value, or a config file path
  goal: 'Explore the login flow and report any bugs',
  target: 'web',
  maxSteps: 8,
  timeoutMs: 600_000,
  interruptSignal: signal,
  onEvent(event) {
    if (event.type === 'explore' && event.progress.phase === 'finding') console.log(event.progress.finding.title);
  },
});

result.status;            // passed | failed | blocked | error | interrupted
result.explore.ended;     // finished | step-limit | time | stuck | aborted
result.explore.findings;  // kind, severity, title, expected, actual, reproduction
result.reportPath;
  • Public wrapper: packages/e2e/src/explore/public-explore.ts wraps the internal explore().
    • Options are an explicit interface, passed through one by one, so CLI-only settings (reporters, output, headed, forceSignal) never reach the explorer.
    • The result is RunResult plus the explore record, with no live config or results.
  • Shared with run(): run() and explore() now share one ownedRun() (packages/e2e/src/run/public-run.ts). It handles the process reservation (RUNNER_BUSY across both), the owned and redacted output, cancellation and onEvent bound into the run's async context, and telemetry (command: 'runner.explore'). run() behaves as before.
  • Config-load output: the internal explore() takes quiet and processOutput. It holds what a config's top-level code prints while the config loads, until the config's secrets are registered, the way run loads its config.
  • Notices: with no notice callback, the notices e2e explore prints before the run starts (the target chosen, a custom executor replaced, a non-default agent) go to the run through tests.notices. The run emits them as run notices during setup, before the plan event, so explore() callers see them in onEvent. The CLI passes its own notice and is unchanged.
  • Errors: options or a config that cannot start an exploration reject before anything starts, as the CLI flags do (INVALID_CONFIG, UNKNOWN_TARGET, MODEL_UNAVAILABLE, and the config load codes). After that, everything resolves with its status in the report.
  • Docs: reference/runner.mdx#explore (with a typechecked example in docs/examples/runner/explore.ts), a pointer from explore.mdx, and the telemetry and security pages. The skill reference (skills/e2e/references/running.md) is synced to the plugin.
  • Type checks: tests/types/sdk-types.ts pins the option and result shapes, and explore is not exported from e2e.

Not in this PR: the CLI's e2e explore still never claims process output, so a config that prints a secret at its top level reaches the terminal unredacted there. e2e run and explore() both redact it. The plumbing now exists, so it's a small follow-up; happy to open it separately.

Verified

Ran it locally: yes, through the built package. There was no live model run, because I don't have a gateway key; the explorations use the repo's scripted model.

  • tests/integration/explore.test.ts, the "explore from e2e/runner" tests, import dist/runner.js and explore the fixture app in Chromium:
    • Finding: a scripted finding fails the run (exit 1). result.explore equals report.run.explore, report.json is written, and onEvent sees started, step-started, finding, step-finished and finished.
    • Cancel: aborting at step-started resolves interrupted (130) with attempt cleanup complete, and the next exploration passes.
    • Rejections and overlap: maxSteps: 0 and an unknown target reject before anything starts. A run() started during an exploration rejects with RUNNER_BUSY.
    • Notices and extra keys: with two targets, the "exploring target" notice arrives as a run notice before plan. output and reporters passed past the type are ignored.
  • Config output in a separate process: a Node script imports e2e/runner and explores with a file config whose top level prints a configured secret. The terminal shows config console.log: <secret:apiToken>. A config that prints and then throws has its output withheld.
    • With the withholdDuring change reverted, both tests fail and the raw token prints:

      without the holdthis branch
      config console.log: explore-output-token-42config console.log: <secret:apiToken>
  • Docs example: docs/examples/runner/explore.ts typechecks against the built packages.

Live run (maintainer, rebased on main)

Script imports explore from built dist/runner.js, runtime E2EConfig with two targets against the testbed bug garden, openai/gpt-6-luna-fast:

0.0 run-started
0.0 notice run exploring target "web"; pass --target to explore another
0.0 plan
3.5 explore step-started Cart and checkout
8.9 explore finding Home greeting exposes an unrendered userName template token
14.3 explore finding Neuromancer catalog listing shows negative stock
22.0 explore finding Checkout newsletter checkbox label misspells "Receive"
30.4 explore finding Order confirmation uses epoch dates and delivery precedes order
36.7 explore finished
37.5 run-finished
RESULT exitCode 1, status failed, ended finished, 4 findings, keys exitCode,status,report,reportPath,aiTracePath,explore
  • Abort 5s into a step: run-interrupted, resolves 130 / interrupted, ended: aborted, partial finding kept, app port released, caller keeps running.
  • Rejections: maxSteps: 99 -> INVALID_CONFIG, target: 'nope' -> UNKNOWN_TARGET, agent: 'ghost' -> INVALID_CONFIG, all isE2EError.
  • Credential password absent from every file under the output dir; video.webm written.