Test Discovery slow for large TestCaseSource and TestCaseAttribute

I am currently using version 3.15.1 of the test adapter in VS 2019 Community edition (16.2.5). .NET Standard 4.5.2 for the code.

I have a solution with >80,000 tests. 2 test projects have about ~40,000 each. There are many tests that have several hundred test cases (either from TestCaseAttribute or TestCaseSource and a class building the test cases). When I go to run my tests, the actual test run is only a few minutes - but the discovery phase takes up to 20 minutes to find/build all test cases. While I understand that constructing large TestCaseSource and parsing hundreds of TestCaseAttributes will take some time, but 20 minutes seems excessive.

[9/23/2019 9:13:57.068 AM Informational] NUnit Adapter 3.15.1.0: Test execution started
[9/23/2019 9:13:57.108 AM Informational] Running selected tests in C:\Users\cidth\Documents\GitHub\CreatureGen\CreatureGen.Tests.Integration.Stress\bin\Debug\CreatureGen.Tests.Integration.Stress.dll
[9/23/2019 9:13:57.308 AM Informational] NUnit Adapter 3.15.1.0: Test execution started
[9/23/2019 9:13:57.351 AM Informational] Running selected tests in C:\Users\cidth\Documents\GitHub\CreatureGen\CreatureGen.Tests.Integration.IoC\bin\Debug\CreatureGen.Tests.Integration.IoC.dll
[9/23/2019 9:13:57.528 AM Informational]    NUnit3TestExecutor converted 3 of 3 NUnit test cases
[9/23/2019 9:13:57.863 AM Informational]    NUnit3TestExecutor converted 26 of 26 NUnit test cases
[9/23/2019 9:13:58.347 AM Informational] NUnit Adapter 3.15.1.0: Test execution started
[9/23/2019 9:13:58.404 AM Informational] Running selected tests in C:\Users\cidth\Documents\GitHub\CreatureGen\CreatureGen.Tests.Integration\bin\Debug\CreatureGen.Tests.Integration.dll
[9/23/2019 9:13:58.592 AM Informational] NUnit Adapter 3.15.1.0: Test execution complete
[9/23/2019 9:13:59.374 AM Informational]    NUnit3TestExecutor converted 757 of 757 NUnit test cases
[9/23/2019 9:13:59.589 AM Informational] NUnit Adapter 3.15.1.0: Test execution complete
[9/23/2019 9:14:02.972 AM Informational] NUnit Adapter 3.15.1.0: Test execution started
[9/23/2019 9:14:03.208 AM Informational] Running selected tests in C:\Users\cidth\Documents\GitHub\CreatureGen\CreatureGen.Tests.Unit\bin\Debug\CreatureGen.Tests.Unit.dll
[9/23/2019 9:14:03.758 AM Informational] NUnit Adapter 3.15.1.0: Test execution started
[9/23/2019 9:14:04.043 AM Informational] Running selected tests in C:\Users\cidth\Documents\GitHub\CreatureGen\CreatureGen.Tests.Integration.Tables\bin\Debug\CreatureGen.Tests.Integration.Tables.dll
[9/23/2019 9:14:11.565 AM Informational] NUnit Adapter 3.15.1.0: Test execution complete
[9/23/2019 9:20:06.993 AM Informational]    NUnit3TestExecutor converted 34144 of 34144 NUnit test cases
[9/23/2019 9:25:08.348 AM Informational]    NUnit3TestExecutor converted 45509 of 45509 NUnit test cases
[9/23/2019 9:32:16.820 AM Informational] NUnit Adapter 3.15.1.0: Test execution complete

I did read over in #448 that the adapter scans for things in serial, and twice - which would certainly be a source of the slowdown. If I set the test runner to use parallel runs (as the task suggests), that does help with the speed, but it is still very slow to do a “Run All”.

The repo where this occurs can be found here. Simple check out the branch, open the solution, build, and click “Run All”. There will be test failures, but the time it takes for test discovery is the concern.

I can run the tests in the console runner, and the test discovery is much faster, which also matches up with #448 .

[CharliePoole]

@aolszowka If you want to do this just to run a timing test, then create a text file with the full names of all the tests to run and use the --testlist option. The console will create the same filter that is created by the adapter for those names.

OTOH, if you are doing this for any other reason than as an experiment, I’d suggest using the time to fix the adapter. 😄

[marcw3]

hi @OsirisTerje and @CharliePoole @cidthecoatrack I know this issue has been discussed at length my issue is due to the fact i’m using VS2022 and using the Test explorer to debug my tests cases. My test cases are using TestCaseSource also. Currently I have 108 test cases and its taking over 9 min for test discovery to happen. I’m stuck on how to debug my test without facing this issue or when its time to run it through devops. I’ve tried different settings on Nunit tips and trick and haven’t found anything viable. I’m currently running Nunit 3.13.3 and Nunit3 test adapter 4.5.0. Any direction would be appreciated