Testing: pytest should be running multiple nodes in one single process

Sorry if I missed an existing feature request with my searches I couldn’t find one.

Hello,

When selecting multiple nodes to run tests, one would/could expect all of them to be run in one single process instantiation, for the following reasons:

• performances: preparing some resources (data, loading modules on a big codebase…) may take some time and it is possible to have this time done once for all instead of once for each
• reporting: one may want to get the coverage report generated by a selection of test nodes and coverage can be consolidated automatically with python-coverage instead of having to manually set coverage file option and then consolidate several files into one
• isolation: it happens, despite efforts, that some tests don’t behave the same when run together instead of one by one. To reproduce isolation issues that may be encountered later in CI, it’d be useful to be able to run in same conditions.

I’ve specified pytest in this issue although I think it may be valid for other runners, because I’m only using pytest and I know it’s possible to specify several tests in command-line invocation.

Thinking about the case where people may expect parallel runs and consider current state as a feature, I think they should rather consider options like pytest-xdist instead.

I dug in the code, and, basically, it would mean that instead of having runTests do an await of delegated calls to runTest for each node, it may directly invoke pytest with all nodes in arguments. Maybe even, if a parent and a child node are selected together only parent node could be passed to invocation, to avoid user mistakes leading to confusion.

In case this feature is accepted and needs external contribution, I may be able to code it

Thank you 🙂

[lachaib]

Yes I get the same because explorer sees 2 tests at the same line (happens also when a test is a method of a class that gets inherited) And so 2 nodes, and so running 2 parallel runs…

[schlamar]

Not sure if this is the exact same issue and if this will be fixed with the linked ticket. I noticed some strange behavior with unexpected parallel test execution.

Minimal pytest example:

import time
import pytest

@pytest.mark.parametrize("data", (1, 2))
def test_parallel(data):
    time.sleep(15)

Running this with the editor icon beside the line number it spawns two Python processes for the 2 tests (you can verify this with Process Explorer for example or just by checking the Python test output). This behavior with parallel execution is unexpected as this might break tests if shared resources are involved (in my case access to shared memory).

Running the test in the testing panel it spawns only one Python test process as expected.