PR #1206 - tester-army/e2e

What and why

Adds filters to the config: test filters that run before the tests and leave out the ones a pull request can't affect. Most pull requests touch one area, and running the whole suite on each one is slow and costs money.
decisionFilter({ model }) in @e2e-dev/decision asks a decision model, such as Jev, whether the diff can change what each test checks, and whether it can break the setup that saves the test's session. It leaves out a test only when both are rated under 20% likely affected.
Filters run only with --changed-since <ref>, the same flag name as Jest's --changedSince. Unlike Playwright's --only-changed, which runs changed test files and the test files that import a changed file, the change here includes app source, and a filter decides what it can affect. CI passes it on pull requests: npx e2e run ${{ github.event_name == 'pull_request' && format('--changed-since origin/{0}', github.base_ref) || '' }} with fetch-depth: 0. main, the merge queue, and local runs without it run every test. Core reads nothing from the CI environment: the flag alone decides.
Core rules, which no filter can change:
- a --last-failed rerun is never filtered, so e2e run || e2e run --last-failed can't go green on a failure a filter left out;
- with no file changed since the merge base, no filter runs;
- an empty or --led ref fails with INVALID_CHANGED_SINCE before anything runs, and so does a ref git can't measure from (unknown, or no merge base in a shallow checkout) when a filter would run, exit 2;
- --changed-since with no filters configured warns that every test runs;
- a filter that throws, returns something invalid, or runs past 5 minutes leaves nothing out, with a warning.
The decision filter's own rules (a custom filter gets every test and decides for itself):
- a changed test file or setup file, or either's stored screenshots (<file>-snapshots/), runs its tests without a question, and the note counts them;
- a change to a file named e2e.config.{ts,mts,js,mjs} or to a lockfile of any package manager (npm, pnpm, yarn, bun, deno, CocoaPods, SwiftPM, Gradle, Bundler, Poetry, uv, Go, Cargo, Composer, pub, Mix, NuGet, Nix, Bazel) leaves nothing out when it is in the project's directory, inside it, or above it (the workspace root). A sibling project's or a docs folder's goes to the model.
e2e list claims and redacts the process output as run does, since filters and model SDKs run in it, and --reporter json keeps stdout for the document.
pull_request_target: the docs recipe passes the flag on pull_request only, and the docs say never to pass it on pull_request_target, where a fork's diff would reach the model in a run that holds the repository's secrets.
The change is every file that differs from the merge base of the ref and HEAD, uncommitted and untracked files included. The decision filter reads only what the commits change; an uncommitted or untracked file goes to the model by name only, so a token a CI step writes into the checkout never reaches it.
This repo's CI uses it on the testbed step: spec.yml gets fetch-depth: 0, AI_GATEWAY_API_KEY, and --changed-since origin/<base> on pull requests.
The tradeoff: a PR's testbed check now runs only what Jev keeps. The merge queue and main still run every test, so the merge gate stays deterministic. This is noted in AGENTS.md.

filters: [decisionFilter({ model: gateway.decisionModel('typesafe-ai/jev'), providerOptions: { gateway: { zeroDataRetention: true } } })],

A custom filter is { name, filter }. The contract is TestFilter in packages/e2e/src/types.ts:1501; each candidate carries setupFile when a setup test saves its session:

export const areas = {
  name: 'areas',
  filter(tests, { changes, signal, redact }) {
    return { exclude: { [testId]: 'the change reaches none of its areas' }, note: 'optional one-liner' };
  },
} satisfies TestFilter;

flowchart LR
  A[select: flags, then --shard] --> B{--changed-since, not --last-failed, a file changed?}
  B -- no --> R[run every selected test]
  B -- yes --> E[filters in config order, each judges what the last kept]
  E --> F[run what is left; left-out tests report as skipped]

The decision filter

What it sends:
- the diff of changes.files only, base to HEAD (git diff -M --irreversible-delete, so renames and deletions cost a header), then each file no commit changes by name, never its contents;
- the test files and the setup files, keyed by their path from the repository root as the diff names them.
- Secrets are redacted from the diff, the file contents, the file paths, and the questions with context.redact, which masks every fixed secret and credential password and any run of 8 or more of their characters, so a multi-line key split across diff lines is masked too.
Questions: one per test, and one per setup file ("can the diff break this setup or the session it saves?"). A test is left out only when both answers are under 20%.
Batching: files are packed into calls that fit Jev's 32k-token state limit, counted in UTF-8 bytes, at most 200 questions per call (a larger file is split across calls), 8 calls at once.
When it leaves tests unjudged, they run:
- a failed call leaves its own tests in, and after 3 failures no more calls start;
- a diff over 60 kB leaves nothing out, with a note.
When no call answers (no key, as on a fork's pull request), or git or a file read fails, the filter throws. Core fails it open: warning: decision failed, so it leaves nothing out: <reason>, outcome: "failed", every test runs.
providerOptions goes with every call, as on decisionExecutor.
Spend: the note reports it, (11 calls, 358.8k tokens, $0.0139). Cost comes from the AI Gateway, using the list price on your own provider key.
Narrowing the change, always running smoke tests, or running everything on an infra/ change is a few-line wrapper around decision.filter, shown in docs/filters.mdx, not an option.

Where it shows

Output	What it shows
Terminal	`decision kept 7 of 77 tests, left out 70 (1 call, 17.6k tokens, $0.0007)`, then a `Left out` row in the summary
PR comment, `summary.md`	`e2e: 7 passed, 70 left out by filters`, the change line, one line per filter, a folded table of left-out tests with reasons
`report.json`	`run.filters` (`outcome` when a filter failed or timed out), `run.changes` (`base` and `files` once git measured the change, else `skipped` alone), `skip.filter` (schema and both fixtures updated)
`junit.xml`	left-out tests as skipped cases
`e2e list --changed-since origin/main`	the same decision, without starting an app

Releases

e2e: patch.
@e2e-dev/github: patch.
@e2e-dev/decision: minor with a Release: line, and its e2e peer becomes >=0.21.3.
New error code: INVALID_CHANGED_SINCE.
New flag: --changed-since.

Verified

Ran it locally: yes

Testbed: built CLI, real Jev through the AI Gateway, a branch that breaks the todo counter (basics.mjs, one line), e2e run --changed-since main --reporter list:

ℹ judging 1 file changed since 3f2c3da
ℹ decision: asking typesafe-ai/jev about 78 tests in 14 files
 ✓ filter "decision" done 874ms
ℹ decision kept 6 of 78 tests, left out 72 (1 call, 18.8k tokens, $0.0007)
...
      Tests  4 failed | 2 passed (6)
   Left out  72 tests by decision

A run of the same branch without the flag fails the same 4 tests.

Large suite: the testbed's tests copied 20 times (280 files, 1540 tests), e2e list --changed-since big-main. It took 2s, and all 80 copies of the 4 failing tests were kept:

decision: asking typesafe-ai/jev about 1540 tests in 280 files
decision kept 132 of 1540 tests, left out 1408 (11 calls, 358.8k tokens, $0.0139)

A login form label change (Password to Passphrase), which only the admin setup test touches. e2e list --changed-since main, 3 runs each. With the setup question, the dashboard admin tests are kept; with setupFile stripped (the old behavior), Jev left them out at 8-13%, so the broken setup never runs. The real run with the filter fails the setup and the one other failing test, the same failures as a run without the flag:

Config	`authenticated session reaches the dashboard directly`	`signing out invalidates the session`
with the setup question	kept (3/3)	kept (3/3)
`setupFile` stripped	left out, 8-11%	left out, 9-11%

Flag-only, on the label branch with an uncommitted probe line appended to the PR's tracked basics.mjs and an untracked file, through a config whose model logs every call's input. The model got the committed diff of basics.mjs without the probe line, and the untracked file by name only:

judging 2 files changed since 3f2c3da
decision kept 5 of 78 tests, left out 73 (1 call, 18.9k tokens, $0.0007)

diff --git a/apps/testbed/app/pages/basics.mjs b/apps/testbed/app/pages/basics.mjs
+         <label for="password">Passphrase</label>
diff --git a/apps/testbed/r6-untracked.txt b/apps/testbed/r6-untracked.txt
not committed, contents not read

The other paths, with the built CLI:

Command	Result
`e2e list` with filters configured	no filter line, no git call; `report.json` has no `run.changes`
`GITHUB_EVENT_NAME=pull_request` with an event payload, no flag	every test, nothing printed: no detection any more
`GITHUB_ACTIONS=true GITHUB_EVENT_NAME=pull_request_target e2e list --changed-since main` (after the flag-only cut)	judged like any run: `judging 3 files changed since 3f2c3da`
`e2e list --changed-since nope`	`INVALID_CHANGED_SINCE: --changed-since "nope" names no commit in this checkout; fetch it, such as with fetch-depth: 0 on actions/checkout`, exit 2
`--changed-since main` with no filters configured	`warning: --changed-since: the config has no filters, so every test runs`
`e2e run --changed-since=-x --last-failed`, no filters	`INVALID_CHANGED_SINCE: --changed-since must name a git ref, such as origin/main; got "-x"`, exit 2
a filter that throws, before one that keeps smoke tests	`warning: boom failed, so it leaves nothing out: kaboom`; `report.json` records `outcome: "failed"` and the next filter still judges all 78
a filter that leaves every test out	`areas left out all 78 tests`, no app started, exit 0, `e2e: no tests selected, 78 left out by filters`
`--last-failed --pass-with-no-tests`	`run.changes` is `{ "skipped": "--last-failed" }`
`e2e list --no-filter`	`error: unknown option '--no-filter'`

An untracked file as the only change, e2e list --changed-since HEAD. The new file breaks the todo count, and Jev kept the 6 todo tests:

decision kept 6 of 77 tests, left out 71 (1 call, 17.5k tokens, $0.0007)

This PR's own check job passes --changed-since origin/main from spec.yml. The PR edits the testbed config, so the decision filter leaves nothing out, as designed (apps/testbed/e2e.config.ts changed, so the decision filter left nothing out).
After moving the lockfile and pin rules into the decision filter, on the label branch:
- an untracked pnpm-lock.yaml: decision kept all 78 tests (pnpm-lock.yaml changed, so the decision filter left nothing out), no call;
- a committed edit to tests/board.e2e.ts: decision: 3 tests run without asking: the test file, setup file, or stored screenshots changed, then asking typesafe-ai/jev about 75 tests in 13 files, kept 8 of 78 (the 3 board tests and the 5 dashboard tests the label change reaches).
Without AI_GATEWAY_API_KEY, as on a fork's pull request, the decision filter fails: the run prints warning: decision failed, so it leaves nothing out: ... with the gateway's authentication error, and every test runs.
A 14-line PEM key set as a secret, split across diff lines by edits, through the logging config: 0 fragments of it reached the model.
A run that stops after the filters but before the first test (Ctrl-C) still records the left-out tests in the report, the events, and the outcome.
Gates on the rebased head: pnpm check, e2e 3790 tests, @e2e-dev/decision 208, @e2e-dev/github 38, @e2e-dev/mobile 350, and CI green.

Docs cuts from the shortening pass: on docs/filters.mdx, "many test files per call and up to 8 calls at once" is cut as an internal detail; it stays in AGENTS.md for contributors.

Round 8 review (5 subagents: core and decision code-quality reviews, an adversarial bug hunt, git and environment edge cases, live Jev CLI runs). Fixed: the docs areas example left out a test whose own file changed (a false green); lockfile and config names matched anywhere in the repo, so a docs example's e2e.config.ts ran every test; e2e list printed a secret a filter or the config logged and broke --reporter json; :/message refs were rejected; the docs said a git failure warns when it is INVALID_CHANGED_SINCE; a partial cost sum was shown as the total; the gate module folded into filterSelection. Held up:
- git layouts: shallow (depth 1, deepened, unshallowed), detached HEAD, worktree, submodule, monorepo subdirectory, symlinked and wrong-case project paths, names with spaces, quotes, newlines, leading dashes, renames, deletes, binaries, mode changes, untracked and ignored files, a merge commit, GitLab merged-results, a hung git (30 s message), no git, not a repo, unrelated histories;
- Linux in Docker (node:24, node:22-bookworm) and Node 22.23.2 and 24.19.0 on macOS;
- 12 live Jev scenarios on the testbed: the counter break (7 of 78 kept, 4 failed, || --last-failed stays red), shards 1/2 and 2/2, junit, report.json schema-valid, summary.md and the GitHub reporter, list vs run, no key and invalid key (warning, all run), everything left out (exit 0, no app), Ctrl-C (130 in 0.03 s), timeout via the harness option, both docs examples, the SDK, --watch through a pty, lockfile and test edits, uncommitted secrets never sent;
- after the scope fix: a root Cargo.lock leaves nothing out, a sibling apps/other/pnpm-lock.yaml goes to Jev (kept 5 of 78).

Not in this PR:

A context.configFile, so a config loaded with --config e2e.ci.config.ts would also run every test; today it goes to the model.
The terminal's Duration and startedAt start after the filters, as they do after engine prepare.
The filter's spend is not added to run.usage or to the run's overall spend line. That needs a usage field on TestFilterResult.
A diff over 60 kB still runs every test. Splitting it across calls, and a sharper question, wait for an eval on PRs from other repos.

Checklist

Changeset added (pnpm changeset) for any change to a published package, or the change is testbed, docs, or CI only.
Breaking change: title carries !, the changeset body names what breaks and what replaces it, and the deprecation warning landed a window earlier (stability policy in CONTRIBUTING.md). Not breaking.
Docs updated in the same PR: the docs/**/*.mdx page for the behavior, and skills/e2e/ if the skill describes it.
Contract change: emitted .d.ts reviewed, tests/types/sdk-types.ts updated; wire change edits the schema and both fixtures.
Engine contract change (e2e/engine): changesets for e2e, @e2e-dev/web, and @e2e-dev/mobile. JSDoc only: a run with nothing to run calls no prepare, which neither engine relies on.