Wrong fixture/conftest being used

I discovered a weird issue when running int tests on our Jenkins node at work, which I cannot reproduce inside a dev/vagrant VM using the same versions. The behaviour described below is only on the Jenkins node, but I don’t really know where to go next in working out why it’s happening. The original errors were in a way more complex setup, but I’ve managed to reduce this to a fairly minimal example. I doubt anyone else will be able to repro this in isolation, but i’m hoping someone can help me with where to look next to work out what’s going on.

Minimal example file structure:

integration_tests/init.py (empty)

integration_tests/aa/init.py (empty)

integration_tests/aa/conftest.py

import pytest
@pytest.fixture
def url():
    return "1"

integration_tests/aa/test_x.py

def test_url(url):
    assert url == "1"

integration_tests/aa_foo/init.py (empty)

integration_tests/aa_foo/conftest.py

import pytest
@pytest.fixture
def url():
    return "2"

integration_tests/aa_foo/test_x.py

def test_url(url):
    assert url == "2"

So we have 2 packages, each containing a conftest defining a fixture url and each containing a test.

07:51:50 py.test integration_tests/
07:51:50 ============================= test session starts ==============================
07:51:50 platform linux -- Python 3.5.1, pytest-3.2.3, py-1.4.34, pluggy-0.4.0
07:51:50 rootdir: /var/lib/jenkins/workspace/adminweb-int-test, inifile:
07:51:50 collected 2 items
07:51:50 
07:51:50 integration_tests/aa/test_x.py .
07:51:50 integration_tests/aa_foo/test_x.py F
07:51:50 
07:51:50 =================================== FAILURES ===================================
07:51:50 ___________________________________ test_url ___________________________________
07:51:50 
07:51:50 url = '1'
07:51:50 
07:51:50     def test_url(url):
07:51:50 >       assert url == "2"
07:51:50 E       AssertionError: assert '1' == '2'
07:51:50 E         - 1
07:51:50 E         + 2
07:51:50 
07:51:50 integration_tests/aa_foo/test_x.py:3: AssertionError
07:51:50 ====================== 1 failed, 1 passed in 0.04 seconds ======================

What we see is that when aa_foo/test_x.py executes, it’s using the wrong url fixture - it’s using the fixture from aa rather than aa_foo. From investigation, it seems to be somehow related to aa and aa_foo sharing a common root name (aa). Longer versions of this name (originally these dirs were named create and create_with_stuff shows the same issue. I shortened the name repeatedly, and aa is the sortest name that shows the issue - if I change the package names to a and a_foo, it starts working again.

As said at the start, it works fine in a local development (vagrant) VM. Both cases (broken jenkins and working dev VM) are running Ubuntu 14.04 and python 3.5, and I’ve also created a new virtual env that only contains/installs py.test 3.2.3 to rule out other packages affecting this.

Are there any known bugs/issues in pytest that can cause this sort of behaviour? The original issue was found in a complex nest of parameterised fixtures defined at different levels, and I had originally assumed the bug was in that nest, but distilling it down to this simple example rules all the complexity out. It seems like pytest is simply using the wrong fixture.

Any help or guidance anyone can provide getting to the bottom of this would be very much appreciated!

[tom-dalton-fanduel]

I think I’ve found the cause of my woes, but I’ll need someone to confirm if this is expected behaviour or a bug. I suspect (hope) it’s the latter!

The problem seems to come down to this method: https://github.com/pytest-dev/pytest/blob/master/_pytest/fixtures.py#L1133

which:

• takes a list of possible fixtures and a node (test file) where the fixture name is used
• returns the list of fixtures that the node can ‘see’

The way it decides if the node can see the fixture is a simple nodeid.startswith(fixturedef.baseid).

This means that in the case we have a test node like integration_tests/aa_foo/test_x.py, then it (I think incorrectly) thinks that the fixture from integration_tests/aa is a match - even though in fact this is a sibling package, rather than a parent package.

I’ll make a minimal example repo to show the behaviour.

[tom-dalton-fanduel]

Closing - Fixed by #2862