Tests not being discovered since 2019.4.0

Environment data

• VS Code version: 1.33.1
• Extension version (available under the Extensions sidebar): 2019.4.11987
• OS and version: Windows 10
• Python version (& distribution if applicable, e.g. Anaconda): 3.6.6 in my virtualenv
• Type of virtual environment used (N/A | venv | virtualenv | conda | …): virtualenv
• Relevant/affected Python packages and their versions: tests?

Expected behaviour

Discover Python unit tests should pick up unit tests (excluding ones I want ignored), mark the unit tests in the source file with the ‘debug test|run test’ annotations, and running unit tests should only run my tests. This was working prior to yesterday.

Actual behaviour

Discover unit tests includes directories I’ve asked to be ignored in python.testing.pyTestArgs, unit test source files are not marked with annotations, and all tests run. Also I’m not seeing the results of Discover Unit Tests anywhere after run it.

Steps to reproduce:

1. My workspace settings file has these settings:

``

Logs

Output for Python in the Output panel (View→Output, change the drop-down the upper-right of the Output panel to Python)

		"python.linting.ignorePatterns": [
			".vscode/*.py",
			"**/site-packages/**/*.py",
			"cps_python_packages",
			"tests",
			"context.py"
		],
		"python.linting.pylintArgs": [
			"--rcfile", 
			"C:\\Users\\me\\source\\repos\\mason\\.pylintrc"
		],
		"python.testing.pyTestArgs": [
			"--ignore=${workspaceFolder}/mason/cps_python_packages/cps_datamodel/tests"
		],

and cps_python_packages shouldn’t be linted or tested, but is being linted and tested. I have no user settings affecting this.

Output from Console under the Developer Tools panel (toggle Developer Tools on under Help; turn on source maps to make any tracebacks be useful by running Enable source map support for extension debugging)

None relevant so far as I can tell

[brettcannon]

I wanted to give some transparency about this whole situation and how we got here as I know some people are a bit frustrated at the moment with our pytest support has been fluctuating a bit over the last few releases.

We used to use a hack to discover tests in pytest and the pytest team specifically asked us to stop (and rightfully so) 😁 . Pytest didn’t have any stability guarantees for the hack we were using, hence why the last few releases there has been skew for what tests we discovered in the extension (totally our fault for using the hack BTW, it’s just the usual software issue of having the time to do the proper fix versus patching what we had).

And so in this release we finally got around to using pytest’s plug-in system for test discovery which is a more proper way. Unfortunately there seems to be a an issue yet again in terms of what pytest reports and what we used to discover using the hack (and I’m not placing blame on pytest, just that discovery is not universally broken, just that it’s hard to pin down). Add to that the fact that pytest is very flexible in its discovery and it makes finding reproducers a bit hard without repos we can clone (thanks to those providing such repos!). We did increase our testing of test discovery as well, so I would like to think this isn’t due to sloppiness on our part as much as the complexity of the whole situation as people’s testing setups are not always simple.

Regardless of everything I have said above, we’re obviously sorry this has caused trouble for folks and we are actively trying to figure out what’s going on.

[DonJayamanne]

Hi everyone:

• It looks like the problems are mostly related to conflicting pytest plugins. So for those that reported a problem, please reply with which pytest plugins you are using. Thanks!
• We will be publishing a new version of the extension tomorrow with the old behavior. Hopefully this should resolve the issues you’ve experienced. In the interim, you can download & install the pre-release version extension from here and try it now
• We have updated the development version of the extension to log the details of the failure. Please could you download & install the development version of the extension from here and try it now. Test discovery will still fail. However this time, the details of the failure will be logged in the Python Test Log output panel. Please provide the details of the failure.

Once again, we apologize for the inconvenience caused by this issue.