feat(runner): expose explore() through e2e/runner
Closes #1196.
e2e/runner gets explore(), the counterpart of e2e explore that run() is to e2e run.
import type { E2EConfig } from 'e2e';
import { explore } from 'e2e/runner';
const result = await explore({
cwd: projectDirectory,
config, // an E2EConfig value, or a config file path
goal: 'Explore the login flow and report any bugs',
target: 'web',
maxSteps: 8,
timeoutMs: 600_000,
interruptSignal: signal,
onEvent(event) {
if (event.type === 'explore' && event.progress.phase === 'finding') console.log(event.progress.finding.title);
},
});
result.status; // passed | failed | blocked | error | interrupted
result.explore.ended; // finished | step-limit | time | stuck | aborted
result.explore.findings; // kind, severity, title, expected, actual, reproduction
result.reportPath;
- Public wrapper:
packages/e2e/src/explore/public-explore.tswraps the internalexplore().- Options are an explicit interface, passed through one by one, so CLI-only settings (
reporters,output,headed,forceSignal) never reach the explorer. - The result is
RunResultplus theexplorerecord, with no liveconfigorresults.
- Options are an explicit interface, passed through one by one, so CLI-only settings (
- Shared with
run():run()andexplore()now share oneownedRun()(packages/e2e/src/run/public-run.ts). It handles the process reservation (RUNNER_BUSYacross both), the owned and redacted output, cancellation andonEventbound into the run's async context, and telemetry (command: 'runner.explore').run()behaves as before. - Config-load output: the internal
explore()takesquietandprocessOutput. It holds what a config's top-level code prints while the config loads, until the config's secrets are registered, the wayrunloads its config. - Notices: with no
noticecallback, the noticese2e exploreprints before the run starts (the target chosen, a custom executor replaced, a non-default agent) go to the run throughtests.notices. The run emits them asrunnotices during setup, before theplanevent, soexplore()callers see them inonEvent. The CLI passes its ownnoticeand is unchanged. - Errors: options or a config that cannot start an exploration reject before anything starts, as the CLI flags do (
INVALID_CONFIG,UNKNOWN_TARGET,MODEL_UNAVAILABLE, and the config load codes). After that, everything resolves with its status in the report. - Docs:
reference/runner.mdx#explore(with a typechecked example indocs/examples/runner/explore.ts), a pointer fromexplore.mdx, and the telemetry and security pages. The skill reference (skills/e2e/references/running.md) is synced to the plugin. - Type checks:
tests/types/sdk-types.tspins the option and result shapes, andexploreis not exported frome2e.
Not in this PR: the CLI's e2e explore still never claims process output, so a config that prints a secret at its top level reaches the terminal unredacted there. e2e run and explore() both redact it. The plumbing now exists, so it's a small follow-up; happy to open it separately.
Verified
Ran it locally: yes, through the built package. There was no live model run, because I don't have a gateway key; the explorations use the repo's scripted model.
tests/integration/explore.test.ts, the "explore from e2e/runner" tests, importdist/runner.jsand explore the fixture app in Chromium:- Finding: a scripted finding fails the run (exit 1).
result.exploreequalsreport.run.explore,report.jsonis written, andonEventseesstarted,step-started,finding,step-finishedandfinished. - Cancel: aborting at
step-startedresolvesinterrupted(130) with attempt cleanupcomplete, and the next exploration passes. - Rejections and overlap:
maxSteps: 0and an unknown target reject before anything starts. Arun()started during an exploration rejects withRUNNER_BUSY. - Notices and extra keys: with two targets, the "exploring target" notice arrives as a
runnotice beforeplan.outputandreporterspassed past the type are ignored.
- Finding: a scripted finding fails the run (exit 1).
- Config output in a separate process: a Node script imports
e2e/runnerand explores with a file config whose top level prints a configured secret. The terminal showsconfig console.log: <secret:apiToken>. A config that prints and then throws has its output withheld.-
With the
withholdDuringchange reverted, both tests fail and the raw token prints:without the hold this branch config console.log: explore-output-token-42config console.log: <secret:apiToken>
-
- Docs example:
docs/examples/runner/explore.tstypechecks against the built packages.
Live run (maintainer, rebased on main)
Script imports explore from built dist/runner.js, runtime E2EConfig with two targets against the testbed bug garden, openai/gpt-6-luna-fast:
0.0 run-started
0.0 notice run exploring target "web"; pass --target to explore another
0.0 plan
3.5 explore step-started Cart and checkout
8.9 explore finding Home greeting exposes an unrendered userName template token
14.3 explore finding Neuromancer catalog listing shows negative stock
22.0 explore finding Checkout newsletter checkbox label misspells "Receive"
30.4 explore finding Order confirmation uses epoch dates and delivery precedes order
36.7 explore finished
37.5 run-finished
RESULT exitCode 1, status failed, ended finished, 4 findings, keys exitCode,status,report,reportPath,aiTracePath,explore
- Abort 5s into a step:
run-interrupted, resolves 130 /interrupted,ended: aborted, partial finding kept, app port released, caller keeps running. - Rejections:
maxSteps: 99->INVALID_CONFIG,target: 'nope'->UNKNOWN_TARGET,agent: 'ghost'->INVALID_CONFIG, allisE2EError. - Credential password absent from every file under the output dir;
video.webmwritten.