Broken namespace packages corner-case? (py3.5)

I have hit a complicated corner case on py3.5 regarding namespace packages on code that used to work before. I suspect it is a bug in a recent setuptools, but I may be wrong… tox is also involved, but I doubt it is at fault.

I have reproduced the whole thing in IwanVosloo/setuptools-namespaces

The gist of it is: code in one module in a namespace package cannot import code from another module in a different setuptools distribution that shares the same namespace package.

I have the following directory structure:

 ├── makeit.sh
 ├── nspace-three
 │   ├── nspace
 │   │   ├── __init__.py
 │   │   └── three
 │   │       ├── __init__.py
 │   │       └── morestuff.py
 │   └── setup.py
 └── nspace-two
     ├── MANIFEST.in
     ├── nspace
     │   ├── __init__.py
     │   └── two
     │       ├── __init__.py
     │       └── stuff.py
     ├── setup.py
     ├── tmp
     └── tox.ini
 

nspace-two and nspace-three are different Distributions sharing a namespace package (nspace), and nspace-two depends on nspace-three.

The module nspace-two/nspace/two/stuff.py starts off with an “import pkg_resources” [1] before it imports something from nspace.three.

If you run tox, nspace-two/tox.ini runs, as a test: python -c 'from nspace.three.morestuff import *' (and this always works) and then: python -c 'from nspace.two.stuff import *' (which breaks on py3.5 but works on 2.7)

If I comment out the import in [1] above - it also works.

If I instruct tox to changedir away from nspace-two directory before doing those imports - it also works.

python: 3.5.2 virtualenv: 15.1.0 pip: 9.0.1 setuptools: 32.3.0

[IwanVosloo]

Thanks @jaraco

We have wrestled with the issue a bit more and have learnt some more about it.

Firstly, this is a strange (yet probably common) scenario: we are running code from the current directory of the source code of a package, but that package is installed (via pip) AS WELL. This seems to confuse setuptools somewhat, because it picks up ./nspace as well as the installed site-packages version of nspace. (./nspace only contains two, whereas the version in site-packages contains three as well.)

Secondly, we realised that this is related to setuptools’ handling of PEP420 implicit namespaces.

According to PEP420, namespace packages should not have __init__.py files at all. We’re on Python 3.5 with this issue, thus, according to PEP420, our namespace packages should not have __init__.py files. This example had __init__.py files that contained calls to .declare_namespace(). If we remove the relevant __init__.py files, the problem goes away.

What causes the issue is:

There are two versions of the nspace package: ./nspace (which is part of the source code of nspace-two) and …site-packages/nspace

When importing nspace.three, somehow the ./nspace is found first, and inside it it finds an init.py. According to PEP420 then, if a package has an __init__.py in it the package is treated as a normal package - NOT a namespace package.

I am not sure how one would handle this scenario. Our code base runs on Python 2.7 as well and I’m not sure if one still needs __init__.py files in that case, since PEP420 is Python >= 3.3. Perhaps a dependency on new enough setuptools would solve it?

On the other hand, the situation where you run, eg “python ./setup.py” in the source code tree of a package which also happens to be installed in site-packages is kindof weird. Yet, I think it often happens. For example, when people run tests against installed packages: python ./setup.py test.

[daa]

@IwanVosloo @tlandschoff-scale May you please check if you have this problem with latest setuptools? Recently my pull request fixing similar issue was merged, I hope it should help with this issue too.