Discover PyTest tests only when necessary

Environment data

• VS Code version: 1.22.2 (1.22.2) 3aeede733d9a3098f7b4bdc1f66b63b0f48c1ef9
• Extension version (available under the Extensions sidebar): 2018.3.1
• OS and version: OSX 10.13.4
• Python version (& distribution if applicable, e.g. Anaconda): 3.6.4
• Type of virtual environment used (N/A | venv | virtualenv | conda | …): conda
• Relevant/affected Python packages and their versions: n/a

Actual behavior

Test discovery automatically runs after saving edits of a test file.

Although this behavior sounds natural, it causes a spike of energy consumption on every save of an edited test file. One might just want to save it without intending to run test immediately afterwards, e.g. for linting. The fan of my computer starts howling after a few saves.

In OSX activity monitor, energy impact changes from ~1 to ~50 after saving an edited test file, and the energy impact stays around 50 for about 3 seconds. I have 22 tests in the file I saved and 245 tests in total.

Related to issue #1474 and #1132.

Expected behavior

Only run test discovery after giving a command to run or clicking Run Test or Debug Test.

Steps to reproduce:

1. Edit a test file.
2. Save the edited test file.
3. Go to step 1 for a few more times until the fan starts howling. This takes 5 times on my computer.

Logs

Output for Python in the Output panel (View→Output, change the drop-down the upper-right of the Output panel to Python)

============================= test session starts ==============================
platform darwin -- Python 3.6.4, pytest-3.4.2, py-1.5.2, pluggy-0.6.0 -- /XXXXXXXX
cachedir: .pytest_cache
Matplotlib: 2.2.2
Freetype: 2.8.1
rootdir: /XXXXXX, inifile: setup.cfg
plugins: xdist-1.22.2, mpl-0.9, forked-0.2, cov-2.5.1
collecting ... collected 245 items

<<Collected tests>>

========================= no tests ran in 2.89 seconds =========================

Output from Console under the Developer Tools panel (toggle Developer Tools on under Help)

<<Empty>>

Question

Any pointer on how test discovery works in vscode-python?

[brettcannon]

This was implemented in the latest release: https://github.com/Microsoft/vscode-python/blob/master/CHANGELOG.md#enhancements

[brettcannon]

Providing a feature to disable automatic test discovery seems reasonable.