Pipenv failed to build wheels (PEP 517)

Given the following:

$ python3 -V
Python 3.9.1

$ pipenv --version
pipenv, version 2020.11.15

$ pip --version
pip 20.3.3

Let us consider mkdir test && cd test the following minimal Pipfile

[[source]]
url = "https://pypi.org/simple"
verify_ssl = true
name = "pypi"

[packages]
pandas = "*"

Running a simple pipenv install generates a mile long error message whose last bits are

  ...
  ----------------------------------------
  ERROR: Failed building wheel for numpy
Failed to build numpy
ERROR: Could not build wheels for numpy which use PEP 517 and cannot be installed directly
...
...
pipenv.patched.notpip._internal.exceptions.InstallationError: Command errored out with exit status 1: /~/.local/share/virtualenvs/test-mmObi3rW/bin/python /usr/local/lib/python3.9/site-packages/pipenv/patched/notpip install --ignore-installed --no-user --prefix /private/var/folders/vw/67twknb114j86_qdm17qgrk40000gn/T/pip-build-env-lje2m4uc/overlay --no-warn-script-location --no-binary :none: --only-binary :none: -i https://pypi.org/simple -- setuptools wheel 'Cython>=0.29.21,<3' 'numpy==1.15.4; python_version=='"'"'3.6'"'"' and platform_system!='"'"'AIX'"'"'' 'numpy==1.15.4; python_version=='"'"'3.7'"'"' and platform_system!='"'"'AIX'"'"'' 'numpy==1.17.3; python_version=='"'"'3.8'"'"' and platform_system!='"'"'AIX'"'"'' 'numpy==1.16.0; python_version=='"'"'3.6'"'"' and platform_system=='"'"'AIX'"'"'' 'numpy==1.16.0; python_version=='"'"'3.7'"'"' and platform_system=='"'"'AIX'"'"'' 'numpy==1.17.3; python_version=='"'"'3.8'"'"' and platform_system=='"'"'AIX'"'"'' 'numpy; python_version>='"'"'3.9'"'"'' Check the logs for full command output.

This problem has been touched already for instance here, here and here concering pip. Apparently the suggestion is to explicitly include the flag --no-use-pep517 which however does not work in conjunction with pipenv, producing pipenv.vendor.requirementslib.exceptions.RequirementError: Failed parsing requirement from '--no-use-pep517'.

It seems there is also a general problem with the version 2020.11.15, I do not exactly understand whether related or not. If instead of pandas one installs pipenv install request (or any other package not containing particular dependencies) the installation works fine, which makes me think the problem is indeed the building of the numpy wheels.

What is the right procedure in this case to just proceed with the installation?

---

$ pipenv --support

Pipenv version: '2020.11.15'

Pipenv location: '/usr/local/lib/python3.9/site-packages/pipenv'

Python location: '/usr/local/opt/python@3.9/bin/python3.9'

Python installations found:

• 3.9.1: ~/.local/share/virtualenvs/test-mmObi3rW/bin/python3
• 3.9.1: ~/.local/share/virtualenvs/test-mmObi3rW/bin/python3.9
• 3.9.1: ~/.local/share/virtualenvs/test-mmObi3rW/bin/python3
• 3.9.1: ~/.local/share/virtualenvs/test-mmObi3rW/bin/python3.9
• 3.9.1: /usr/local/bin/python3
• 3.9.1: /usr/local/bin/python3.9
• 3.8.2: /usr/bin/python3
• 2.7.16: /usr/bin/python2
• 2.7.16: /usr/bin/python2.7

PEP 508 Information:

{'implementation_name': 'cpython',
 'implementation_version': '3.9.1',
 'os_name': 'posix',
 'platform_machine': 'x86_64',
 'platform_python_implementation': 'CPython',
 'platform_release': '20.2.0',
 'platform_system': 'Darwin',
 'platform_version': 'Darwin Kernel Version 20.2.0: Wed Dec  2 20:39:59 PST '
                     '2020; root:xnu-7195.60.75~1/RELEASE_X86_64',
 'python_full_version': '3.9.1',
 'python_version': '3.9',
 'sys_platform': 'darwin'}

System environment variables:

• TERM_SESSION_ID
• SSH_AUTH_SOCK
• LC_TERMINAL_VERSION
• COLORFGBG
• ITERM_PROFILE
• XPC_FLAGS
• PWD
• SHELL
• __CFBundleIdentifier
• LC_CTYPE
• TERM_PROGRAM_VERSION
• TERM_PROGRAM
• PATH
• LC_TERMINAL
• COLORTERM
• COMMAND_MODE
• TERM
• HOME
• TMPDIR
• USER
• XPC_SERVICE_NAME
• LOGNAME
• ITERM_SESSION_ID
• __CF_USER_TEXT_ENCODING
• SHLVL
• OLDPWD
• P9K_TTY
• ZSH
• PAGER
• LESS
• LSCOLORS
• P9K_SSH
• EDITOR
• LANG
• MANPAGER
• FZF_DEFAULT_OPTS
• FZF_DEFAULT_COMMAND
• PIP_DISABLE_PIP_VERSION_CHECK
• PYTHONDONTWRITEBYTECODE
• PIP_PYTHON_PATH
• PIPENV_ACTIVE
• VIRTUAL_ENV
• PS1
• _
• PIP_SHIMS_BASE_MODULE
• PYTHONFINDER_IGNORE_UNSUPPORTED

Pipenv–specific environment variables:

• PIPENV_ACTIVE: 1

Debug–specific environment variables:

• PATH: ~/.local/share/virtualenvs/test-mmObi3rW/bin:~/.cargo/bin:/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin:/Library/TeX/texbin:/usr/local/MacGPG2/bin:/usr/local/opt/fzf/bin
• SHELL: /bin/zsh
• EDITOR: vim
• LANG: en_US.UTF-8
• PWD: ~/test
• VIRTUAL_ENV: ~/.local/share/virtualenvs/test-mmObi3rW

---

Contents of Pipfile (‘~/test/Pipfile’):

[[source]]
url = "https://pypi.org/simple"
verify_ssl = true
name = "pypi"

[packages]
pandas = "*"

[gennaro-tedesco]

Can you install it via pipenv run pip install --no-use-pep517 numpy?

doing so produces the same error stated above, whose complete stacktrace is

Failed to build numpy
Installing collected packages: numpy
    Running setup.py install for numpy ... error
    ERROR: Command errored out with exit status 1:
     command: ~/.local/share/virtualenvs/test-mmObi3rW/bin/python -u -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/private/var/folders/vw/67twknb114j86_qdm17qgrk40000gn/T/pip-install-r4nvpr4k/numpy/setup.py'"'"'; __file__='"'"'/private/var/folders/vw/67twknb114j86_qdm17qgrk40000gn/T/pip-install-r4nvpr4k/numpy/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' install --record /private/var/folders/vw/67twknb114j86_qdm17qgrk40000gn/T/pip-record-agkzau_0/install-record.txt --single-version-externally-managed --compile --install-headers /Users/gennarotedesco/.local/share/virtualenvs/test-mmObi3rW/include/site/python3.9/numpy
         cwd: /private/var/folders/vw/67twknb114j86_qdm17qgrk40000gn/T/pip-install-r4nvpr4k/numpy/
    Complete output (41 lines):
    Running from numpy source directory.

    Note: if you need reliable uninstall behavior, then install
    with pip instead of using `setup.py install`:

      - `pip install .`       (from a git repo or downloaded source
                               release)
      - `pip install numpy`   (last NumPy release on PyPi)


    Cythonizing sources
    numpy/random/_bounded_integers.pxd.in has not changed
    Processing numpy/random/_philox.pyx
    Traceback (most recent call last):
      File "/private/var/folders/vw/67twknb114j86_qdm17qgrk40000gn/T/pip-install-r4nvpr4k/numpy/tools/cythonize.py", line 59, in process_pyx
        from Cython.Compiler.Version import version as cython_version
    ModuleNotFoundError: No module named 'Cython'

    During handling of the above exception, another exception occurred:

    Traceback (most recent call last):
      File "/private/var/folders/vw/67twknb114j86_qdm17qgrk40000gn/T/pip-install-r4nvpr4k/numpy/tools/cythonize.py", line 235, in <module>
        main()
      File "/private/var/folders/vw/67twknb114j86_qdm17qgrk40000gn/T/pip-install-r4nvpr4k/numpy/tools/cythonize.py", line 231, in main
        find_process_files(root_dir)
      File "/private/var/folders/vw/67twknb114j86_qdm17qgrk40000gn/T/pip-install-r4nvpr4k/numpy/tools/cythonize.py", line 222, in find_process_files
        process(root_dir, fromfile, tofile, function, hash_db)
      File "/private/var/folders/vw/67twknb114j86_qdm17qgrk40000gn/T/pip-install-r4nvpr4k/numpy/tools/cythonize.py", line 188, in process
        processor_function(fromfile, tofile)
      File "/private/var/folders/vw/67twknb114j86_qdm17qgrk40000gn/T/pip-install-r4nvpr4k/numpy/tools/cythonize.py", line 64, in process_pyx
        raise OSError('Cython needs to be installed in Python as a module')
    OSError: Cython needs to be installed in Python as a module
    Traceback (most recent call last):
      File "<string>", line 1, in <module>
      File "/private/var/folders/vw/67twknb114j86_qdm17qgrk40000gn/T/pip-install-r4nvpr4k/numpy/setup.py", line 508, in <module>
        setup_package()
      File "/private/var/folders/vw/67twknb114j86_qdm17qgrk40000gn/T/pip-install-r4nvpr4k/numpy/setup.py", line 488, in setup_package
        generate_cython()
      File "/private/var/folders/vw/67twknb114j86_qdm17qgrk40000gn/T/pip-install-r4nvpr4k/numpy/setup.py", line 285, in generate_cython
        raise RuntimeError("Running cythonize failed!")
    RuntimeError: Running cythonize failed!
    ----------------------------------------
ERROR: Command errored out with exit status 1: ~/.local/share/virtualenvs/test-mmObi3rW/bin/python -u -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/private/var/folders/vw/67twknb114j86_qdm17qgrk40000gn/T/pip-install-r4nvpr4k/numpy/setup.py'"'"'; __file__='"'"'/private/var/folders/vw/67twknb114j86_qdm17qgrk40000gn/T/pip-install-r4nvpr4k/numpy/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' install --record /private/var/folders/vw/67twknb114j86_qdm17qgrk40000gn/T/pip-record-agkzau_0/install-record.txt --single-version-externally-managed --compile --install-headers ~/.local/share/virtualenvs/test-mmObi3rW/include/site/python3.9/numpy Check the logs for full command output.

Looking around some more on the internet I managed to make it work by means of the mysterious and obscure command export SYSTEM_VERSION_COMPAT=1, found somehow accidentally here.

–no-use-pep517 is not the correct path to resolving the issue, Pip will also remove the support in the future

What then exactly is the solution to the problem? More generally, is pipenv keeping up with pip and python updates and versionings? If so, there must be a way to install packages without having to perform workarounds as above, must it not?

[jagu2012]

(Slightly OT please forgive the noise. For those finding this thread looking for numpy/scipy solutions)

As of 4th August 2021 with a M1 Macbook Pro with Big Sur v11.4 (20F71) Python 3.9.6 (Homebrew):

Numpy just works (installs numpy v1.21.1):

pip install numpy

Scipy works with a combination of build flags and and libraries from homebrew (installs scipy v1.7.0):

brew install openblas
export OPENBLAS=$(brew --prefix openblas)
export CFLAGS="-falign-functions=8 ${CFLAGS}"
pip install Cython pybind11 pythran
pip install --no-use-pep517 scipy