Using fixtures in the collection phase

TL;DR: Is it possible to use the result of the doctest_namespace fixture in the collection phase? If not, what is the name of first hook after the autouse fixtures have been called?

---

I’m the maintainer of pytest-sphinx, which is a plugin that adds support for running tests defined in doctest related rst directives. Those directives are implemented in the sphinx.ext.doctest extension.

One cool feature of sphinx.ext.doctest is that directives can be skipped conditionally by specifying an option inside the body of the directive. Something like

.. testsetup::
   :skipif: pd is None

   data = pd.Series([42])

If the result of the skipif expression is a true value, the directive is omitted from the test run just as if it wasn’t present in the file at all.

In sphinx.ext.doctest the globals used for evaluating the skipif expressions are taken from the evaluation of the code in the doctest_global_setup configuration value (in conf.py), e.g.,

doctest_global_setup = """
    try:
        import pandas as pd
    except ImportError:
        pd = None
"""

Instead of evaluating a code-string in pytest-sphinx, I think it makes more sense to use pytest’s doctest_namespace fixture:

try:
     import pandas as pd
except ImportError:
     pd = None

@pytest.fixture(autouse=True)
def add_pd(doctest_namespace):
    doctest_namespace['pd'] = pd

---

How should pytest-sphinx handle the creation of test items in the following example?

.. testcode::
   :skipif: six.PY3

   data = pd.Series([42])

.. testcode::
   :skipif: six.PY2

   data = pd.Series([42])

.. testoutput::
   :skipif: six.PY3

   xxxxxx

.. testoutput::
   :skipif: six.PY2

   yyyyyy

Every testcode directive should be (regardless of whether the directive is skipped or not) part of a pytest.Item. Since the skipif expressions can’t be evaluated in the collection-phase AFAIK, the unskipped testoutput directive, belonging to the unskipped testcode directive, has to be determined as soon as the doctest-namespace is known. This is tricky, but probably it is doable, right?

Do you have a better idea for supporting the skipif option in pytest-sphinx? Probably it would be a lot easier if doctest-namespace were not used, because then we could evaluate the skipif expressions at collection-time.

[SalmonMode]

I don’t mean to gatekeep, and I feel like my ideal definition of fixtures falls within your own.

The distinction I draw is based in practicality. A test is only as good as it is repeatable. If your tests don’t know what they’re going to do before they start running, and they’re beholden to the state of the system as it already is, then they aren’t completely repeatable because they aren’t in control.

Pytest’s fixtures as they stand now, from my pov, are a perfect system for laying out the steps of a repeatable test, and for describing the resources those steps depend on. They describe the essence of what a given test is; not a series of steps to find out what tests can be done.

Again, that’s just my pov. My goal isn’t to force others to write tests a certain way. It’s to encourage them to write tests that are repeatable and in complete control of the SUT.

I’m not really gonna complain if such a feature as this is implemented, since I can just not use that feature. But I did want to throw my 2 cents out there.

[RonnyPfannschmidt]

This is simply for me, fixtures are a life cycle management and dependency injection system, its unavailability at collection time means people are forced to implement poor sidechannels

The only real reason they are not avaliable is how things initially grew, once runtestprotocol is broken down there are no more technical reasons not to do it