feat: Add filter support for TestAdapter

Merged
FI
1y ago
+267-2

This PR add --filter parameter support for dotnet test.
It intended to resolve #2767 issue (and some part of issue #2662)

What's changed in this PR.

1. Add benchmark filter logics
Add following files to support benchmark filtering.

  • TestCaseFilter.cs
  • LoggerHelper.cs

Note: These files are based on xUnit v2 TestAdapter Code (License: Apache 2.0)

2. Modify VSTestAdapter.cs
Add benchmark filter logics to DiscoverTests and RunTests

3. BenchmarkEnumerator.cs
Remove code for Debug configuration.
(By #2774 changes. It can load DLLs that are build with Debug configurations)

4. Disable TestTfmsInParallel MSBuild property
It's enabled by default on .NET 9 or later.
But it's not expected multiple benchmarks are executed in parallel.

I've added code to disable this setting to following files.

  • BenchmarkDotNet.Samples.csproj
  • BenchmarkDotNet.Samples.FSharp.csproj
  • BenchmarkDotNet.TestAdapter/build/BenchmarkDotNet.TestAdapter.props

5. Temporary comment out IntroVisualStudioProfiler benchmark
Because it cause errors when benchmarks are enumerated/evaluated twice. (#2758)

Tasks that is not scope of this PR

  • Benchmark progress is not shown when running benchmark with dotnet test by default
    -> It need to specify --logger:"console;verbosity=normal" explicitly
  • DisplayName filter is same as FullyQualifiedName
    -> Currently there is no attribute that can be used for DisplayName

What's Tested

Currently it's hard to write unit tests that use TestAdapter.
So it need to run tests manually.

TestExplorer

1. All benchmarks(net462/net80) are displayed when using Release configuration.

  1. It can run single selected test and multiple selected tests.
  2. When running multiple TFMs benchmarks. It's sequentially executed. (Not executed in parallel)

dotnet test

  1. Verify default behavior is not changed when run command without --filter parameter

    dotnet test -c Release -tl:off --no-build --framework net8.0 --logger:"console;verbosity=detailed" --list-tests
    dotnet test -c Release -tl:off --no-build --framework net8.0 --logger:"console;verbosity=detailed"

  2. Filter by partial query works as expected.

    dotnet test -c Release --filter IntroBasic -tl:off --no-build --framework net8.0 --logger:"console;verbosity=detailed" --list-tests
    dotnet test -c Release --filter IntroBasic -tl:off --no-build --framework net8.0 --logger:"console;verbosity=detailed"

  3. Filter by property value works as expected.

    dotnet test -c Release --filter FullyQualifiedName=BenchmarkDotNet.Samples.IntroBasic.Sleep -tl:off --no-build --framework net8.0 --logger:"console;verbosity=detailed" --list-tests
    dotnet test -c Release --filter DisplayName=BenchmarkDotNet.Samples.IntroBasic.Sleep -tl:off --no-build --framework net8.0 --logger:"console;verbosity=detailed" --list-tests

    dotnet test -c Release --filter FullyQualifiedName=BenchmarkDotNet.Samples.IntroBasic.Sleep -tl:off --no-build --framework net8.0 --logger:"console;verbosity=detailed"
    dotnet test -c Release --filter DisplayName=BenchmarkDotNet.Samples.IntroBasic.Sleep -tl:off --no-build --framework net8.0 --logger:"console;verbosity=detailed"

  4. Filter by Category works as expected

    dotnet test -c Release --filter Category=Fast -tl:off --no-build --framework net8.0 --logger:"console;verbosity=normal" --list-tests
    dotnet test -c Release --filter Category=Fast -tl:off --no-build --framework net8.0 --logger:"console;verbosity=normal"

  5. Verify it can run .NET462 benchmarks also

    dotnet test -c Release --filter IntroBasic -tl:off --no-build --framework net462 --logger:"console;verbosity=detailed" --list-tests
    dotnet test -c Release --filter IntroBasic -tl:off --no-build --framework net462 --logger:"console;verbosity=detailed"