setup.py should not put runtime dependencies in setup_requires as it breaks pip dependency management

When installing pylxd with pip, pip invokes its setup.py with the egg_info command to retrieve egg metadata about pylxd (cf. https://pip.readthedocs.io/en/stable/reference/pip_install/#build-system-interface).

This has the side effect of installing pylxd dependencies which currently breaks because of an incompatibility between current requests 2.18.4 and urllib3 1.23.

Example with current tarball from https://pypi.org/project/pylxd/#files:

# wget -q https://files.pythonhosted.org/packages/a0/5e/af099af60d089b28df6b550d34c1e807ce4e3906d257744f55573c4d3cbb/pylxd-2.2.6.tar.gz                            
# tar xf pylxd-2.2.6.tar.gz
# cd pylxd-2.2.6/
# virtualenv test
New python executable in /home/pdecat/workspaces/tmp/pylxd-2.2.6/test/bin/python
Installing setuptools, pip, wheel...done.
# ./test/bin/pip freeze
# ./test/bin/python setup.py egg_info
Installed /home/pdecat/workspaces/tmp/pylxd-2.2.6/.eggs/requests_unixsocket-0.1.5-py2.7.egg
Searching for requests!=2.8.0,>=2.5.2
Reading https://pypi.org/simple/requests/
Downloading https://files.pythonhosted.org/packages/49/df/50aa1999ab9bde74656c2919d9c0c085fd2b3775fd3eca826012bef76d8c/requests-2.18.4-py2.py3-none-any.whl#sha256=6a1b267aa90cac58ac3a765d067950e7dbbf75b1da07e895d1f594193a40a38b
Best match: requests 2.18.4
Processing requests-2.18.4-py2.py3-none-any.whl
Installing requests-2.18.4-py2.py3-none-any.whl to /home/pdecat/workspaces/tmp/pylxd-2.2.6/.eggs
writing requirements to /home/pdecat/workspaces/tmp/pylxd-2.2.6/.eggs/requests-2.18.4-py2.7.egg/EGG-INFO/requires.txt

Installed /home/pdecat/workspaces/tmp/pylxd-2.2.6/.eggs/requests-2.18.4-py2.7.egg
Searching for pbr>=1.8
Reading https://pypi.org/simple/pbr/
Downloading https://files.pythonhosted.org/packages/b3/5d/c196041ffdf3e34ba206db6d61d1f893a75e1f3435699ade9bd65e089a3d/pbr-4.0.4-py2.py3-none-any.whl#sha256=3747c6f017f2dc099986c325239661948f9f5176f6880d9fdef164cb664cd665
Best match: pbr 4.0.4
Processing pbr-4.0.4-py2.py3-none-any.whl
Installing pbr-4.0.4-py2.py3-none-any.whl to /home/pdecat/workspaces/tmp/pylxd-2.2.6/.eggs

Installed /home/pdecat/workspaces/tmp/pylxd-2.2.6/.eggs/pbr-4.0.4-py2.7.egg
Searching for urllib3>=1.8
Reading https://pypi.org/simple/urllib3/
Downloading https://files.pythonhosted.org/packages/bd/c9/6fdd990019071a4a32a5e7cb78a1d92c53851ef4f56f62a3486e6a7d8ffb/urllib3-1.23-py2.py3-none-any.whl#sha256=b5725a0bd4ba422ab0e66e89e030c806576753ea3ee08554382c14e685d117b5
Best match: urllib3 1.23
Processing urllib3-1.23-py2.py3-none-any.whl
Installing urllib3-1.23-py2.py3-none-any.whl to /home/pdecat/workspaces/tmp/pylxd-2.2.6/.eggs
writing requirements to /home/pdecat/workspaces/tmp/pylxd-2.2.6/.eggs/urllib3-1.23-py2.7.egg/EGG-INFO/requires.txt

Installed /home/pdecat/workspaces/tmp/pylxd-2.2.6/.eggs/urllib3-1.23-py2.7.egg
Traceback (most recent call last):
  File "setup.py", line 34, in <module>
    pbr=True)
  File "/home/pdecat/workspaces/tmp/pylxd-2.2.6/test/local/lib/python2.7/site-packages/setuptools/__init__.py", line 128, in setup
    _install_setup_requires(attrs)
  File "/home/pdecat/workspaces/tmp/pylxd-2.2.6/test/local/lib/python2.7/site-packages/setuptools/__init__.py", line 123, in _install_setup_requires
    dist.fetch_build_eggs(dist.setup_requires)
  File "/home/pdecat/workspaces/tmp/pylxd-2.2.6/test/local/lib/python2.7/site-packages/setuptools/dist.py", line 514, in fetch_build_eggs
    replace_conflicting=True,
  File "/home/pdecat/workspaces/tmp/pylxd-2.2.6/test/local/lib/python2.7/site-packages/pkg_resources/__init__.py", line 779, in resolve
    raise VersionConflict(dist, req).with_context(dependent_req)
pkg_resources.ContextualVersionConflict: (urllib3 1.23 (/home/pdecat/workspaces/tmp/pylxd-2.2.6/.eggs/urllib3-1.23-py2.7.egg), Requirement.parse('urllib3<1.23,>=1.21.1'), set(['requests']))

As mentioned in several places, runtime dependencies should not be put in setup_requires as it breaks pip’s dependency management:

Warning Finally, beware of the setup_requires keyword arg in setup.py. The (rare) packages that use it will cause those dependencies to be downloaded by setuptools directly, skipping pip’s protections. If you need to use such a package, see Controlling setup_requires.

cf. https://pip.readthedocs.io/en/stable/user_guide/#installation-bundles

Controlling setup_requires

Setuptools offers the setup_requires setup() keyword for specifying dependencies that need to be present in order for the setup.py script to run. Internally, Setuptools uses easy_install to fulfill these dependencies.

pip has no way to control how these dependencies are located. None of the Package Index Options have an effect.

cf. https://pip.readthedocs.io/en/stable/reference/pip_install/#controlling-setup-requires

Related:

https://github.com/pypa/pip/issues/1884 https://github.com/pypa/pip/issues/3691 https://github.com/pypa/pip/issues/4156#issuecomment-290534167

https://github.com/requests/requests/pull/4669 https://github.com/requests/requests/pull/4671

[ajkavanagh]

@pdecat Just to let you know that 2.2.7 has been released with the relevant patches in it. Thanks again.

[ajkavanagh]

@pdecat I’ll push out 2.2.7 tomorrow - I had to do some testing against various older distros and all the lxds!