Astropy test customizations break pytest for affiliated packages

_Moved from https://github.com/astropy/astropy-helpers/issues/258_

Every so often I make the mistake of running pytest (or py.test for old versions) instead of python setup.py test. It seems like this should work, but instead I get errors collecting tests that seem to be related to the astropy helpers. Even though the file doesn’t appear in the traceback below, it seems to come from imports in the conftest.py from the package template.

This also appears to bite unwary users of packages that use the template: http://stackoverflow.com/questions/31410723/error-using-pytest-tutorial

$ pytest
=============================================== test session starts ================================================
platform darwin -- Python 2.7.12, pytest-3.0.3, py-1.4.31, pluggy-0.4.0
rootdir: /Users/jlong/software/webbpsf, inifile: setup.cfg
collected 0 items / 1 errors

====================================================== ERRORS ======================================================
________________________________________________ ERROR collecting  _________________________________________________
../../homebrew/Cellar/python/2.7.12/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/py-1.4.31-py2.7.egg/py/_path/common.py:332: in visit
    for x in Visitor(fil, rec, ignore, bf, sort).gen(self):
../../homebrew/Cellar/python/2.7.12/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/py-1.4.31-py2.7.egg/py/_path/common.py:368: in gen
    if p.check(dir=1) and (rec is None or rec(p))])
../../homebrew/Cellar/python/2.7.12/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/_pytest/main.py:672: in _recurse
    ihook = self.gethookproxy(path)
../../homebrew/Cellar/python/2.7.12/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/_pytest/main.py:575: in gethookproxy
    my_conftestmodules = pm._getconftestmodules(fspath)
../../homebrew/Cellar/python/2.7.12/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/_pytest/config.py:314: in _getconftestmodules
    mod = self._importconftest(conftestpath)
../../homebrew/Cellar/python/2.7.12/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/_pytest/config.py:350: in _importconftest
    self.consider_conftest(mod)
../../homebrew/Cellar/python/2.7.12/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/_pytest/config.py:373: in consider_conftest
    if self.register(conftestmodule, name=conftestmodule.__file__):
../../homebrew/Cellar/python/2.7.12/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/_pytest/config.py:225: in register
    ret = super(PytestPluginManager, self).register(plugin, name)
../../homebrew/Cellar/python/2.7.12/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/_pytest/vendored_packages/pluggy.py:371: in register
    hook._maybe_apply_history(hookimpl)
../../homebrew/Cellar/python/2.7.12/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/_pytest/vendored_packages/pluggy.py:768: in _maybe_apply_history
    res = self._hookexec(self, [method], kwargs)
../../homebrew/Cellar/python/2.7.12/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/_pytest/vendored_packages/pluggy.py:339: in _hookexec
    return self._inner_hookexec(hook, methods, kwargs)
../../homebrew/Cellar/python/2.7.12/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/_pytest/vendored_packages/pluggy.py:334: in <lambda>
    _MultiCall(methods, kwargs, hook.spec_opts).execute()
../../homebrew/Cellar/python/2.7.12/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/_pytest/vendored_packages/pluggy.py:614: in execute
    res = hook_impl.function(*args)
../../homebrew/Cellar/python/2.7.12/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/astropy/tests/pytest_plugins.py:137: in pytest_configure
    if not config.getoption('remote_data'):
../../homebrew/Cellar/python/2.7.12/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/_pytest/config.py:1148: in getoption
    raise ValueError("no option named %r" % (name,))
E   ValueError: no option named u'remote_data'
============================================== pytest-warning summary ==============================================
WC1 None [pytest] section in setup.cfg files is deprecated, use [tool:pytest] instead.
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! Interrupted: 1 errors during collection !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
==================================== 1 pytest-warnings, 1 error in 1.62 seconds ====================================

At @pllim’s request, I tried the last 2.x release of pytest as well:

$ py.test
=============================================== test session starts ================================================
platform darwin -- Python 2.7.12, pytest-2.9.2, py-1.4.31, pluggy-0.3.1
rootdir: /Users/jlong/software/webbpsf, inifile: setup.cfg
collected 0 items / 1 errors

====================================================== ERRORS ======================================================
________________________________________________ ERROR collecting  _________________________________________________
../../homebrew/Cellar/python/2.7.12/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/py-1.4.31-py2.7.egg/py/_path/common.py:332: in visit
    for x in Visitor(fil, rec, ignore, bf, sort).gen(self):
../../homebrew/Cellar/python/2.7.12/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/py-1.4.31-py2.7.egg/py/_path/common.py:368: in gen
    if p.check(dir=1) and (rec is None or rec(p))])
../../homebrew/Cellar/python/2.7.12/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/_pytest/main.py:647: in _recurse
    ihook = self.gethookproxy(path)
../../homebrew/Cellar/python/2.7.12/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/_pytest/main.py:550: in gethookproxy
    my_conftestmodules = pm._getconftestmodules(fspath)
../../homebrew/Cellar/python/2.7.12/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/_pytest/config.py:297: in _getconftestmodules
    mod = self._importconftest(conftestpath)
../../homebrew/Cellar/python/2.7.12/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/_pytest/config.py:333: in _importconftest
    self.consider_conftest(mod)
../../homebrew/Cellar/python/2.7.12/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/_pytest/config.py:356: in consider_conftest
    if self.register(conftestmodule, name=conftestmodule.__file__):
../../homebrew/Cellar/python/2.7.12/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/_pytest/config.py:208: in register
    ret = super(PytestPluginManager, self).register(plugin, name)
../../homebrew/Cellar/python/2.7.12/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/_pytest/vendored_packages/pluggy.py:370: in register
    hook._maybe_apply_history(hookimpl)
../../homebrew/Cellar/python/2.7.12/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/_pytest/vendored_packages/pluggy.py:747: in _maybe_apply_history
    res = self._hookexec(self, [method], kwargs)
../../homebrew/Cellar/python/2.7.12/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/_pytest/vendored_packages/pluggy.py:338: in _hookexec
    return self._inner_hookexec(hook, methods, kwargs)
../../homebrew/Cellar/python/2.7.12/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/_pytest/vendored_packages/pluggy.py:333: in <lambda>
    _MultiCall(methods, kwargs, hook.spec_opts).execute()
../../homebrew/Cellar/python/2.7.12/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/_pytest/vendored_packages/pluggy.py:596: in execute
    res = hook_impl.function(*args)
../../homebrew/Cellar/python/2.7.12/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/astropy/tests/pytest_plugins.py:137: in pytest_configure
    if not config.getoption('remote_data'):
../../homebrew/Cellar/python/2.7.12/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/_pytest/config.py:1061: in getoption
    raise ValueError("no option named %r" % (name,))
E   ValueError: no option named u'remote_data'
============================================= 1 error in 1.47 seconds ==============================================

I also confirmed that removing the conftest.py I mentioned above resolves the issue, which implicates the from astropy.tests.pytest_plugins import * line. Hence the moved issue.

[josePhoenix]

Well, it’s from this commit: https://github.com/astropy/astropy/pull/2044/files#diff-e602ffb35fbd67477e5543c9cc3846d4R132

And most likely because no default was specified (see http://docs.pytest.org/en/latest/writing_plugins.html?highlight=getoption#_pytest.config.Config.getoption ).

[josePhoenix]

• I can control the version of the pytest executable
• It is a documented way of running the astropy tests
• It is the format used in all the examples: http://docs.pytest.org/en/latest/usage.html
• Not every pytest option is available through Astropy’s setup.py test

For projects I’m part of that use more-or-less “vanilla” PyTest, the instructions are “cd projdir/; pytest”. If questions come up, I can refer to http://docs.pytest.org/.

For affiliated packages, the instructions are “python setup.py test”. That much is equivalently simple. But as soon as you need to go beyond that, all bets are off. The command line arguments are not documented anywhere beyond the level in the python setup.py test --help output. I opened an issue with astropy-helpers because I’m using the package template and the setup.py customizations live in the helpers, and I was told it’s actually in astropy. I guess there’s documentation in the astropy docs (partially reproducing the pytest documentation), but I have yet to meet a casual contributor that reads all the documentation of the project they’re contributing to and all the documentation of its dependencies.

But really, at the end of the day, I’m quite lazy and would simply rather type

pytest ./webbpsf/tests/test_errorhandling.py

than

ASTROPY_USE_SYSTEM_PYTEST=1 python setup.py test -t ./webbpsf/tests/test_errorhandling.py

to accomplish the same thing.