pyproject.toml prevents `pip install --user --editable` (v19.0.3)

Environment

• pip version: pip 19.0.3
• Python version: Python 3.6.7
• OS: Ubuntu bionic 18.04.2 LTS

Description There seems to be no way to install an editable package in userspace with a pyproject.toml file present. At least on Ubuntu, Debian, possibly other systems while not using a Python virtual environment.

Given the following minimal project, both pip install --user -e . and pip install -e . fail if, and only if the pyproject.toml file is present:

$ tree
.
|____setup.py
|____pyproject.toml
|____src
| |______init__.py

$ find . -type f -exec echo -e '-----\n#{}' \; -exec cat {} \;
-----
#./setup.py
#!/usr/bin/env python
from setuptools import setup, find_packages
setup(
    name='nouser',
    version='0.1',
    packages=find_packages('src'),
    package_dir={'src': 'src'},
)
-----
#./pyproject.toml
[build-system]
requires = ['setuptools', 'wheel']
-----
#./src/__init__.py
def oh_hell():
    print('hello')
oh_hell()

Expected behavior

With a pyproject.toml present next to setup.py, either

pip install -e .

or

pip install --user -e .

installs the package in userspace without special privileges and python -c 'import nouser; nouser.oh_hell()' prints hello successfully.

Actual behavior

$ ls
pyproject.toml setup.py  src
$ pip install -e .
Looking in indexes: https://pypi.org/simple/
Obtaining file:///tmp/tmp.I6P01cKtsE
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
    Preparing wheel metadata ... done
Installing collected packages: nouser
  Running setup.py develop for nouser
    Complete output from command /usr/bin/python3 -c "import setuptools, tokenize;__file__='/tmp/tmp.I6P01cKtsE/setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" develop --no-deps:
    running develop
    error: can't create or remove files in install directory
    
    The following error occurred while trying to add or remove files in the
    installation directory:
    
        [Errno 13] Permission denied: '/usr/local/lib/python3.6/dist-packages/test-easy-install-15379.write-test'
    
    The installation directory you specified (via --install-dir, --prefix, or
    the distutils default setting) was:
    
        /usr/local/lib/python3.6/dist-packages/
    
    Perhaps your account does not have write access to this directory?  If the
    installation directory is a system-owned directory, you may need to sign in
    as the administrator or "root" account.  If you do not have administrative
    access to this machine, you may wish to choose a different installation
    directory, preferably one that is listed in your PYTHONPATH environment
    variable.
    
    For information on other options, you may wish to consult the
    documentation at:
    
      https://setuptools.readthedocs.io/en/latest/easy_install.html
    
    Please make the appropriate changes for your system and try again.
    
    
    ----------------------------------------
Command "/usr/bin/python3 -c "import setuptools, tokenize;__file__='/tmp/tmp.I6P01cKtsE/setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" develop --no-deps" failed with error code 1 in /tmp/tmp.I6P01cKtsE/

$ ls
pyproject.toml setup.py  src
$ pip install --user -e .
Looking in indexes: https://pypi.org/simple/
Obtaining file:///tmp/tmp.I6P01cKtsE
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
    Preparing wheel metadata ... done
Installing collected packages: nouser
  Running setup.py develop for nouser
    Complete output from command /usr/bin/python3 -c "import setuptools, tokenize;__file__='/tmp/tmp.I6P01cKtsE/setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" develop --no-deps --user --prefix=:
    usage: -c [global_opts] cmd1 [cmd1_opts] [cmd2 [cmd2_opts] ...]
       or: -c --help [cmd1 cmd2 ...]
       or: -c --help-commands
       or: -c cmd --help
    
    error: option --user not recognized
    
    ----------------------------------------
Command "/usr/bin/python3 -c "import setuptools, tokenize;__file__='/tmp/tmp.I6P01cKtsE/setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" develop --no-deps --user --prefix=" failed with error code 1 in /tmp/tmp.I6P01cKtsE/

But without pyproject.toml:

$ ls
pyproject.toml setup.py  src
$ rm pyproject.toml 
$ ls
setup.py  src
$ pip install --user -e .
Looking in indexes: http://10.17.65.203/root/devel/+simple/, https://pypi.org/simple/
Obtaining file:///tmp/tmp.I6P01cKtsE
Installing collected packages: nouser
  Running setup.py develop for nouser
Successfully installed nouser

Additional Information

Running the listed command manually installs the package without problems:

/usr/bin/python3 -c "import setuptools, tokenize;__file__='/tmp/tmp.I6P01cKtsE/setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" develop --no-deps --user --prefix=
running develop
running egg_info
writing src/nouser.egg-info/PKG-INFO
writing dependency_links to src/nouser.egg-info/dependency_links.txt
writing top-level names to src/nouser.egg-info/top_level.txt
reading manifest file 'src/nouser.egg-info/SOURCES.txt'
writing manifest file 'src/nouser.egg-info/SOURCES.txt'
running build_ext
Creating /home/jan/.local/lib/python3.6/site-packages/nouser.egg-link (link to src)
Adding nouser 0.1 to easy-install.pth file

Installed /tmp/tmp.I6P01cKtsE/src

[cjerdonek]

Your summary seems mostly or roughly correct at points, but I wouldn’t want to say it’s correct without qualification for fear of leading you down the wrong path. For example, here are a couple clarifications (there could be others):

Somewhere the trickery is too elaborate, and that is why installation fails

It’s failing not because it’s too elaborate, but because it wasn’t meant to support this case. It should be failing, but for a different reason. More explanation is below.

Because --no-use-pep517 should cause pyprojefct.toml to be ignored, the installation should succeed when using --no-use-pep517

It’s not that pyproject.toml should be ignored. Rather, --no-use-pep517 should cause the PEP 517 process not to be used in your particular case. (The pyproject.toml file should still be read for the build dependency info.) Again, more below.

To restate some things, PEP 517 introduced a new way of building packages that involves setting up an isolated environment. I would encourage you to take a look: https://www.python.org/dev/peps/pep-0517/

One important aspect missing from your points is any mention of editable mode. It’s important because that’s what your issue involves.

Since PEP 517 doesn’t support editable mode, pip should be erroring out / failing fast if the user tries to use the new PEP 517 style with editable mode (but it’s not failing fast). Currently, I would say whatever it’s doing now is unsupported / undefined, which I believe is the behavior you’re running into.

Feel free to ask specific questions for further clarification.

[cjerdonek]

See also issue #6314 for additional discussion.