Namespace package breaks module imports

Here’s a simple example to reproduce the problem. In https://github.com/matham/pip-nspace-broken there’s a farm package as well as a farm.deps.glew meant to share a namespace with the farm package.

Also included are the wheels built with it. After installing the wheels with pip, I get the following:

$ python
Python 3.4.4 (v3.4.4:737efcadf5a6, Dec 20 2015, 20:20:57) [MSC v.1600 64 bit (AMD64)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> from farm import my_name
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ImportError: cannot import name 'my_name'

If I remove farm.deps.glew I get the following:

$ python
Python 3.4.4 (v3.4.4:737efcadf5a6, Dec 20 2015, 20:20:57) [MSC v.1600 64 bit (AMD64)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> from farm import my_name
Imported farm

This is with pip 9.0.1.

So basically, having a namespace package seems to break the main package.

[ncoghlan]

For this kind of question, it’s important to separate two different concepts:

1. Optional submodules, where a given component installs into a shared package module hierarchy. This doesn’t require namespace packages at all, it just requires what @matham describes in https://github.com/pypa/setuptools/issues/895#issuecomment-268722067

In 3.3+ and when using importlib2 that will work standalone (since __init__.py files become optional), while in 3.2 and earlier it will require farm or farm.deps as a dependency that provides the relevant __init__.py files.

1. Namespace packages, which allow a package to be split across multiple directories by having more than one entry in __path__.

Python 3.3+ treats any directory without an __init__.py file as a potential part of a namespace package automatically, while the declare_namespace functionality in setuptools lets you request the generation of a boilerplate __init__.py that takes care of setting __path__ appropriately based on a search of sys.path.

The fact that these are separate questions means that a non-namespace package may have namespace packages as children, as long as it includes the necessary __path__ expansion code in its __init__.py: https://docs.python.org/3/library/pkgutil.html#pkgutil.extend_path

The trick with doing that is that you don’t tell setuptools that it is a namespace package, you instead publish it as a normal self-contained package that happens to manipulate its own __path__ attribute.

However, if all you want is optional submodules, then you don’t need to use namespace packages at all - just make the submodules depend on the parent package that they install into, and let the parent package take care of installing the necessary __init__.py files.

[jaraco]

@matham: That may work in some scenarios, but will probably not work in cases where farm and farm.deps.glew are not installed to the same location (i.e. one is installed in system site packages and the other in user site packages, or one under develop and the other not, or one or more of the packages are installed as an egg). Also, I’m sort-of surprised that import farm.deps (or anything below it) works at all on Python prior to 3.3, as there’s no module for that package.