A way to specify build-dependencies

Pipenv might need a way to specify built-dependencies and tooling packages that need to be installed in order for pipenv to do it’s job correctly. Even a pyporject.toml is not sufficient, because you’d have to pass pip the option --no-use-pep517 in order to install build dependencies and then have pip build from setup.py the old-fashioned way.

Example

In my company we use a wrapper around setuptools to simplify writing setup.py scripts for our packaging. Let’s call this wrapper stwrapper.

A typical setup.py would look like this:

# setup.py for "package-that-uses-stwrapper-in-its-setup-py"
from stwrapper import setup

setup(
   name='package-that-uses-stwrapper-in-its-setup-py',
   # ... some more arguments
)

Where the function stwrapper.setup parses arguments, does some extra work and then calls the real setuptools.setup. Effectively, the stwrapper package is sort of a built dependency for us.

Lets say we have the following Pipfile:

[dev-packages]
stwrapper = "*"
package-that-uses-stwrapper-in-its-setup-py = { path = ".", editable = true, requires=["stwrapper"] }

and do a

pipenv lock --verbose

then, with the current version of Pipenv, we get:

Traceback:
...
pipenv.patched.notpip._internal.exceptions.InstallationError: Command "python setup.py egg_info" failed with error code 1 in '.'

Because stwarpper is not installed at the time pipenv tries to parse the setup.py of package-that-uses-stwrapper-in-its-setup-py. That is even the case, when stwrapper is installed in the system’s python installation. Pipenv wants to have it inside the new virtual environment it creates. It’s a Catch-22.

Our current workaround is to let pipenv create the virutal environment. Then do a pipenv run pip install stwrapper inside it and then the pipenv lock. That way it works.

Another, better workaround will be possible one https://github.com/pypa/pip/pull/6370 is merged into pip. Then the behavior of pip-option --no-use-pep517 will be to install requirements specified in a pyproject.toml and proceed with pre-pep517 installation using setup.py.

Possible Solutions

1.) Pipenv gets an equivalent to setup_requires in setup files or requires=[...] in pyproject.toml’s [backend] section that is global or we use the existing one that is currently only used for python_version:

   [requires]
   python_version = "2.7"
   stwrapper = "*"

   [packages]
   ...

2.) Pipenv uses the system python’s installation of stwrapper (I’m aware of the --site-packages option but that may have unwanted side effects).
3.) Specify a per-package requires, i.e.

[packages]
mypackage = { version="2.0.1", requires=['stwrapper', 'gitpython'] }

4.) Pass --no-use-pep517 to pip for certain packages (similar syntax to 3.) and use that feature in conjunction with pyproject.toml for these packages

[packages]
mypackage = { version="2.0.1", pip_options=['--no-use-pep517'] }

5.) Specify an order/priority of package installation, so that the dependency is fullfilled when the dependent package is being installed/

Scope

I guess the problem applies to all projects that use non-standard-libaray packages in their setup.py files and/or use setuptools entry points to modify its behavior. You can not put such dependencies in the Pipfile, because they are needed for successfully locking.

Related Issues

• https://github.com/pypa/pipenv/issues/2274
• https://github.com/pypa/pip/pull/6370
• https://github.com/pypa/pipenv/issues/1760

[uranusjr]

I wonder if this can be fixed more simply by passing --no-use-pep517 during the retry phase of installation.

[con-f-use]

Okay, I just learned that the build-system solution, does not work in case of editable installs, unless pip is called with --no-use-pep517.

See PEP 517:

This PEP [517] originally specified another hook, install_editable, to do an editable install (as with pip install -e). It was removed due to the complexity of the topic, but may be specified in a later PEP.

Briefly, the questions to be answered include: what reasonable ways existing of implementing an ‘editable install’? Should the backend or the frontend pick how to make an editable install? And if the frontend does, what does it need from the backend to do so.

So the issue persists. It seems something like this, might be useful:

[dev-packages]
package-that-uses-stwrapper-in-its-setup-py = { path = ".", editable = true, requires=["stwrapper"] }
# or
package-that-uses-stwrapper-in-its-setup-py = { path = ".", editable = true, pip_options=["--no-use-pep517"] }  # needs a pyproject.toml with the build-dependencies and no build-backand