Enable multiple test suites and test configurations in the Test UI

Problem Statements

• P1. Currently VS Code for Python only allows a single test configuration for each test framework. For example, using PyTest, I can apply certain arguments to run whenever PyTest is run through the UI, but if I setup that configuration to exclude certain tests (such as those annotated with “slow”), I have to manually update the arguments in the options window to switch to another configuration.

• P2. There is also no way to setup separate groups of tests and show them in the Test sidebar window as those groups. For example, rather than having a module-based hierarchy, it would be great to have the ability to have them grouped by arbitrary groups of modules/classes/methods and/or based on annotations. Currently one cannot have the tree show different groups of test, such as those annotated as “slow” or “sanity”.

• P3. In addition, any tests that are excluded by the current test configuration cannot be run even when they are directly selected to run as an individual test, because the current configuration causes them to be excluded by the test framework. So that one test will be detected, but then excluded from running. So if I normally want to exclude “slow” tests, that means I cannot even click directly on a slow test and run it directly in the UI.

Current Elements in the UI:

• Existing layout in “Test” sidebar window:
• Existing In-line Test Run / Debug overlay:

Use Cases

Use cases for allowing creation and easy selection of multiple test groupings and configurations:

• UC1 - Running sometimes with coverage, and sometimes not, due to coverage taking longer. Alternately, generating different coverage report types.
• UC2 - Creating longer-run tests (i.e. “slow” or “detailed” tests) that are only run occasionally for final checks, but the length of time makes it difficult to run the standard test suite regularly after small changes (i.e. after it takes more than a few seconds, and especially after 30-60 seconds).
• UC3 - Marking “sanity” tests that are the key tests that should be run all the time (typiclaly with the goal for the test suite taking a few seconds or less)
• UC4 - identifying groups of modules that should be run together when working in a particular module and making changes. The test system often requires either distributing all tests with each module or putting all tests in a package into a single test folder hierarchy, which may not make sense in larger programs. For example, I may want to run detailed tests within the current module, but also run some high-level sanity checks any time I modify within that module. The need is to allow for regrouping of tests in the test sidebar window based on manually selected groups and/or broad rules such as presence or absence of a given test annotation.
• UC5 - sometimes just running the tests for immediate results, other times running an official test suite for final metrics for review/PR before merging. In this case the test configuration could just be launching a command-line script, etc. In fact, every configuration could potentially be just running a specific Python script, perhaps with specific arguments, rather than running the test framework directly.

Requirements

AS A VS Code user writing larger python programs and packages with tests

I NEED a way to create and run different test suites for a given program or package, that are not solely based on modules or test classes, and to run them under varying sets of arguments or test configurations

• I need the ability to run specific test suites/groups that have been created from within the Test sidebar and/or inline test code overlay elements (i.e. potentially adding additional options to the “run test” and “debug test” commands that appear inline in the code)
• I need a way to create different “test configurations” or sets of arguments for launching supported unit test frameworks, which can either be selected directly (say, through a dropdown in the Test sidebar) and/or applied separately to any given test suite.

SO THAT

1. I can setup multiple test configurations that provide a means of selecting the test arguments and/or options to use when running tests through the Test UI.
2. I can setup these test configurations to run particular test types (i.e. tests annotated as slow, sanity, run all tests, etc.) - e.g. in PyTest, generally through @pytest.mark annotation
3. I can select one test configuration of several that have been setup as the current configuration
4. When I run tests in the sidebar window (through the green triangle “play” icons) there are options to use either the default configuration (either of all tests, or the one setup for this particular test suite) or, optionally, run the currently selected test configuration
5. Preferably, though optionally, these configurations could be easily run from the command line as well. e.g. if they were saved as separate config files or scripts that coudl be run as separate scripts or arguments when running the test framework.
6. I am able to run any single test, either in the Test sidebar window, or through the inline code interface, regardless of whether it would run or not run as a part of all tests under the current configuration (e.g. this test is marked a “slow” test, and the current configuration excludes “slow” tests). There are various ways this could be implemented, such as:
   • providing a separate icon for running under a “default” or “single-test” cofiguration selection
   • separate filtering from other configuration arguments - i.e. have a separate config setup for filtering from all other test configs

Concepts

1. Allow a file or some GUI (perhaps directly in the VS Code options) that defines a test configuration. a. Optionally, separate test groupings (i.e. slow, sanity, etc.) from other test options b. Options would include any argument that can be fed to the test framework, such as running coverage, the output type of the coverage, different test framework settings. c. These settings would supersede the current “test arguments” option in VS Code that is present for each of the individual test frameworks, which would be consider the “default” test configuration for all tests.

2. Allow the user to select the current test configuration from a dropdown or similar UI element a. Likely in the sidebar window, perhaps on a separate line just under the header row that says “Python”. b. Such a dropdown is envisioned to work like the Run/Debug dropdown in VS Code where you can either select an existing test configuration or create a new one, through an entry always present at the bottom of the list. c. Alternately, or in addition, there could be an annotation that allows switching configurations that is inline, or at least that is an inline link that opens whereever configurations are selected in the sidebar (or in VS Code options).

3. Reorganize the test hierarchy in the sidebar window based on test configurations options that could use module, test class, and/or annotations to specify how to create the test tree.

   • e.g. Name a group as “sanity”, and put all tests with the “sanity” annotations in it, Name a second group as “slow” and put all tests with the “slow” annotation in it. Have a separate group for “normal”, and finally “other” (any test not showing up elsewhere) and “all” (which includes all discovered tests).
   • This would provide a means of point-and-click to run a specific group of tests that have been defined by the user.

4. Provide a means of applying different default test configurations to different sets of test groups. For example, under concept 3, the user is already defining different test suites or groups of tests, the user should also have some means of changing the default configuration for that group (or maybe the only configuration for that group?)

   • Under this concept, the user could create a test group/suite for all “slow” tests that run without coverage, but then create a second group that had the same tests in it (all the “slow” tests), but run with coverage.
   • Provide a means of running either the configured test configuration for a given test group, or fall back to the default configuration (i.e. add a second “run” icon or link that would run under the overall default test configuration).

Acceptance Criteria

1. The user is able to setup multiple test configurations in terms of the test arguments that will be used when running any tests in the current test framework (unittest, pytest, nose, etc.).

2. User designated test groups a. The user is able to setup multiple test suites or test groups that show up in the Test sidebar window based on:

   1. Manually choosing specific test modules, test classes, and/or test functions/methods
   2. Annotations or some other broader criteria
   3. e.g… run all tests in 2 specific modules, that have the “slow” annotation, or no annotation)

   b. If a test is included in multiple suites, it will show up in each of them c. A separate category shows all tests not included in any other custom test groups, if any are specified (i.e. “other” category) d. A separate category, “all”, always shows all tests in the current module-based hierarchy format

3. The user is able to run or debug any single test directly with either the default test configuration, or a test configuration specifically setup for running individual tests.

4. A given test suite or configuration would allow for running an external script that runs the test framework (with all appropriate arguments), rather than running the framework directly by VS Code.

5. The user is able to select a “current test configuration” that is used instead of the default test configuration, and is used for any case where a group or test doesn’t have a configuration specified

   • Depending on the final design, the user has a means of setting the current test configuration to override all other configurations, or to select to run the current configuration instead of the configuration that would otherwise be run (through an alternate “play” icon on all tests and test groups, for example))

[jarshwah]

My use case is running different subsets of tests with different environment variables or command line arguments (either would be fine, cli arguments preferred).

I have tests structured in the following fashion:

src/tests/unit/...
src/tests/integration/...
src/tests/functional/...

Each require different settings. Subpaths within each folder structure can also require specific settings.

It seems like the launch configuration concept for debugging and tasks would serve for test configurations. I have no idea how that would extend out to the Testing panel or the lenses.

[luabud]

fyi @eleanorjboyd @karthiknadig