fix(decision): leave nothing out when the loaded config changes

Merged
OK
4d ago
+100-16

decisionFilter (#1206, new in #1210) runs every test when a run-all file changes, but it recognized the config only by the names e2e.config.*. A run with --config e2e.ci.config.ts sent a change to that file to the decision model like any other file. In the pre-release bug bash, 3 of 4 decision models left out all 78 testbed tests for a config-only change.

  • e2e hands filters context.configFile, the config the run loaded, relative to changes.root as changes.files spells it (packages/e2e/src/run/test-filters.ts). The directory is resolved on disk and the file keeps its own name, so a symlinked config matches the path git lists. It is undefined for a config object or a config outside the repository.
  • decisionFilter leaves nothing out when that file changes, whatever its name, compared in NFC (packages/decision/src/filter.ts). With an older e2e that does not set the field, it falls back to the name list.
  • configFile is optional on the public TestFilterContext, so a hand-built context in a user's filter tests still compiles.

Not covered: files the config imports still go to the model.

Verified

Ran it locally: yes

  • A scratch git repo with three tests and a custom e2e.ci.config.ts using decisionFilter({ model: gateway.decisionModel('typesafe-ai/jev') }), with only that config changed. Built CLI, e2e list --config e2e.ci.config.ts --changed-since HEAD~1:
mainthis branch
model call, 2 of 3 tests left outno model call, all 3 kept
# main
judging 1 file changed since 42794d4
decision: asking typesafe-ai/jev about 3 tests in 3 files
decision kept 1 of 3 tests, left out 2 (1 call, 867 tokens, <$0.0001)
tests/cart.e2e.ts › cart page shows its heading [web] (left out by decision: 15% likely affected)
tests/login.e2e.ts › login page shows its heading [web]
tests/search.e2e.ts › search page shows its heading [web] (left out by decision: 15% likely affected)

# this branch
judging 1 file changed since 42794d4
decision kept all 3 tests (e2e.ci.config.ts changed, so the decision filter left nothing out)
tests/cart.e2e.ts › cart page shows its heading [web]
tests/login.e2e.ts › login page shows its heading [web]
tests/search.e2e.ts › search page shows its heading [web]
  • The integration test covers configFile for a discovered config, a custom --config name, a project nested in a monorepo, a symlinked config, and a config object (packages/e2e/tests/integration/test-filters.test.ts).