Deprecate test command

I think there’s at least some agreement in #931 that we want to remove the test command. I think we should start by raising deprecation warnings pointing people at tox, the same way we’ve done for the upload and register commands.

The most likely stumbling block that I see is that I think a huge number of people have created their own test command that invokes their preferred test runner, pytest or whatever. Ideally we’d want to get the deprecation warning to them as well. Hopefully most of them are using TestCommand as their base class, though if we want to get really aggressive about it we could try parsing sys.argv directly.

I think we need to warn in the following situations:

1. If the setup.py test command is executed
2. If tests_require is specified
3. If aliases.test is specified in setup.cfg

It’s likely that at least two of these will be specified, but I think two separate warnings would be useful.

CC: @gaborbernat @RonnyPfannschmidt @nicoddemus

[jayvdb]

The problem that I think most people in this thread are having is that there currently is no standard way to execute tests in Python.

setup.py test is that standard way to execute tests. It has been like that for one and a half decades. It has been baked into uni courses and peoples scripts. It is the basis of Django’s manage.py test.

Failing that, unittest test discovery is also standard https://docs.python.org/3/library/unittest.html#unittest-test-discovery

pytest is the very large elephant in the room. A lot of Python direction problems would be solved by declaring that pytest is the new standard (in addition to unittest). Wrap it in a PEP. 😉

setup.py test never really worked all that well to start with.

It works very well for lots of packages.

It doesnt work well for all packages, of course, but that isnt reason to break it for all packages before there is a good replacement. PEP 517 didnt cover test running, but it has laid the groundwork for it, and it wont be long before PEP 517/8 build systems incorporate solutions for that problem and/or a PEP is accepted which does standardised the hooks for test running.

setuptools should keep supporting its existing user-base, keep them working correctly, with all of the features they rely on.

With PEP 517/8, new build systems can flourish. The goal should be for setuptools to be deprecating itself, and the deprecation notice provide a list of all high quality replacement PEP 517/8 build systems which have implemented drop-in solutions for the many varied types of projects which setuptools has supported for the last decade or more. That would be in the spirit of PEP 517, and would avoid setuptools existing near-monopoly implicitly preventing other build systems from being used and supported.

[jayvdb]

This is a great way to kill sdists. (why have sdists if they have the same contents as wheels) How quickly the lesson of sourceforge.net has been forgotten. Or more recently Oracle changing the Term of Use of the Java downloads. IMO setuptools should be moving in the opposite direction, automatically including in sdists the test modules found using test_require or identified using pytest dry-run.

Anyways, unless setuptools also kills its plugin mechanism, someone will recreate the test command as soon as it is removed from setuptools. Maybe that is a good thing. (But if the goal is only to get that stuff out of the setuptools repo because it is hindering setuptools, why not split it off to a separate project now, rather than pretend that tox is a drop in solution)