can no longer pip install numpy & scipy from source at the same time

Environment

• pip version: 19.2.3
• Python version: 2.7.16
• OS: macOS 10.14.6
• pyenv 1.2.13
• setuptools 41.2.0
• wheel 0.33.6

Description

Installing numpy and scipy from source at the same time like this:

pip install numpy==1.14.6 scipy==1.0.1 --no-binary numpy,scipy --force-reinstall

worked fine recently, with these same versions of numpy and scipy, but now it produces a strange error [full error text below]:

RuntimeError: Python version >= 3.5 required.
ERROR: Command errored out with exit status 1: python setup.py egg_info Check the logs for full command output.

Expected behavior Successful installation of numpy and scipy.

How to Reproduce

1. Install pyenv, pyenv-virtualenv, and pyenv-virtualenvwrapper via brew.

2. Install python 2.7.16 via pyenv.

3. Install OpenBLAS 0.3.5 from source into /usr/local/opt/openblas/. Maybe that works now via brew. I don’t know if openblas is needed to reproduce this problem.

4. Create ~/.numpy-site.cfg containing text that points to the openblas installation, e.g.:

   [openblas]
   libraries = openblas
   library_dirs = /usr/local/opt/openblas/lib
   include_dirs = /usr/local/opt/openblas/include
   

5. Run these commands in a new directory:

   pyenv local 2.7.16
   pyenv virtualenv issue
   pyenv local issue
   python -V   # --> Python 2.7.16
   pip install --upgrade pip setuptools virtualenv virtualenvwrapper virtualenv-clone wheel
   pip list
   
       Package           Version
       ----------------- -------
       pbr               5.4.2
       pip               19.2.3
       setuptools        41.2.0
       six               1.12.0
       stevedore         1.30.1
       virtualenv        16.7.4
       virtualenv-clone  0.5.3
       virtualenvwrapper 4.8.4
       wheel             0.33.6
   
   pip install numpy==1.14.6 scipy==1.0.1 --no-binary numpy,scipy --force-reinstall
   

6. This produces the error output, below, in about 8 seconds. That is, it fails during the preparation stage before installing anything.

7. Workaround: Install numpy then scipy:

   pip install numpy==1.14.6 --no-binary numpy --force-reinstall
   pip install scipy==1.0.1 --no-binary scipy --force-reinstall
   

8. Note: We’ve long needed to install numpy before the rest of requirements.txt to avoid an installation dependency error. That error message states that it can’t find numpy.

The present case produces a surprising error saying that it needs Python 3.5+. I don’t know why installing numpy first avoids that scipy installation error. It succeeds if I install numpy then install all requirements.txt including scipy.

Output

DEPRECATION: Python 2.7 will reach the end of its life on January 1st, 2020. Please upgrade your Python as Python 2.7 won't be maintained after that date. A future version of pip will drop support for Python 2.7. More details about Python 2 support in pip, can be found at https://pip.pypa.io/en/latest/development/release-process/#python-2-support
Collecting numpy==1.14.6
  Using cached https://files.pythonhosted.org/packages/86/c2/84dc6f58171bca90326f71e098438b87aa0c0d4a21bceda9caba2af6554e/numpy-1.14.6.zip
Collecting scipy==1.0.1
  Using cached https://files.pythonhosted.org/packages/bd/f4/3882758754dc083fea6ea66a6e8ceef55e7df173d06a12a074612958800f/scipy-1.0.1.tar.gz
    ERROR: Command errored out with exit status 1:
     command: /usr/local/var/pyenv/versions/2.7.16/envs/issue/bin/python2.7 -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/private/var/folders/_b/q30qg_l50b5gvqd8y4_wb9h00000gn/T/pip-install-MCTUmd/scipy/setup.py'"'"'; __file__='"'"'/private/var/folders/_b/q30qg_l50b5gvqd8y4_wb9h00000gn/T/pip-install-MCTUmd/scipy/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' egg_info --egg-base pip-egg-info
         cwd: /private/var/folders/_b/q30qg_l50b5gvqd8y4_wb9h00000gn/T/pip-install-MCTUmd/scipy/
    Complete output (53 lines):
    Traceback (most recent call last):
      File "<string>", line 1, in <module>
      File "/private/var/folders/_b/q30qg_l50b5gvqd8y4_wb9h00000gn/T/pip-install-MCTUmd/scipy/setup.py", line 418, in <module>
        setup_package()
      File "/private/var/folders/_b/q30qg_l50b5gvqd8y4_wb9h00000gn/T/pip-install-MCTUmd/scipy/setup.py", line 414, in setup_package
        setup(**metadata)
      File "/usr/local/var/pyenv/versions/2.7.16/envs/issue/lib/python2.7/site-packages/setuptools/__init__.py", line 144, in setup
        _install_setup_requires(attrs)
      File "/usr/local/var/pyenv/versions/2.7.16/envs/issue/lib/python2.7/site-packages/setuptools/__init__.py", line 139, in _install_setup_requires
        dist.fetch_build_eggs(dist.setup_requires)
      File "/usr/local/var/pyenv/versions/2.7.16/envs/issue/lib/python2.7/site-packages/setuptools/dist.py", line 719, in fetch_build_eggs
        replace_conflicting=True,
      File "/usr/local/var/pyenv/versions/2.7.16/envs/issue/lib/python2.7/site-packages/pkg_resources/__init__.py", line 782, in resolve
        replace_conflicting=replace_conflicting
      File "/usr/local/var/pyenv/versions/2.7.16/envs/issue/lib/python2.7/site-packages/pkg_resources/__init__.py", line 1065, in best_match
        return self.obtain(req, installer)
      File "/usr/local/var/pyenv/versions/2.7.16/envs/issue/lib/python2.7/site-packages/pkg_resources/__init__.py", line 1077, in obtain
        return installer(requirement)
      File "/usr/local/var/pyenv/versions/2.7.16/envs/issue/lib/python2.7/site-packages/setuptools/dist.py", line 786, in fetch_build_egg
        return cmd.easy_install(req)
      File "/usr/local/var/pyenv/versions/2.7.16/envs/issue/lib/python2.7/site-packages/setuptools/command/easy_install.py", line 679, in easy_install
        return self.install_item(spec, dist.location, tmpdir, deps)
      File "/usr/local/var/pyenv/versions/2.7.16/envs/issue/lib/python2.7/site-packages/setuptools/command/easy_install.py", line 705, in install_item
        dists = self.install_eggs(spec, download, tmpdir)
      File "/usr/local/var/pyenv/versions/2.7.16/envs/issue/lib/python2.7/site-packages/setuptools/command/easy_install.py", line 890, in install_eggs
        return self.build_and_install(setup_script, setup_base)
      File "/usr/local/var/pyenv/versions/2.7.16/envs/issue/lib/python2.7/site-packages/setuptools/command/easy_install.py", line 1158, in build_and_install
        self.run_setup(setup_script, setup_base, args)
      File "/usr/local/var/pyenv/versions/2.7.16/envs/issue/lib/python2.7/site-packages/setuptools/command/easy_install.py", line 1144, in run_setup
        run_setup(setup_script, args)
      File "/usr/local/var/pyenv/versions/2.7.16/envs/issue/lib/python2.7/site-packages/setuptools/sandbox.py", line 253, in run_setup
        raise
      File "/usr/local/var/pyenv/versions/2.7.16/lib/python2.7/contextlib.py", line 35, in __exit__
        self.gen.throw(type, value, traceback)
      File "/usr/local/var/pyenv/versions/2.7.16/envs/issue/lib/python2.7/site-packages/setuptools/sandbox.py", line 195, in setup_context
        yield
      File "/usr/local/var/pyenv/versions/2.7.16/lib/python2.7/contextlib.py", line 35, in __exit__
        self.gen.throw(type, value, traceback)
      File "/usr/local/var/pyenv/versions/2.7.16/envs/issue/lib/python2.7/site-packages/setuptools/sandbox.py", line 166, in save_modules
        saved_exc.resume()
      File "/usr/local/var/pyenv/versions/2.7.16/envs/issue/lib/python2.7/site-packages/setuptools/sandbox.py", line 141, in resume
        six.reraise(type, exc, self._tb)
      File "/usr/local/var/pyenv/versions/2.7.16/envs/issue/lib/python2.7/site-packages/setuptools/sandbox.py", line 154, in save_modules
        yield saved
      File "/usr/local/var/pyenv/versions/2.7.16/envs/issue/lib/python2.7/site-packages/setuptools/sandbox.py", line 195, in setup_context
        yield
      File "/usr/local/var/pyenv/versions/2.7.16/envs/issue/lib/python2.7/site-packages/setuptools/sandbox.py", line 250, in run_setup
        _execfile(setup_script, ns)
      File "/usr/local/var/pyenv/versions/2.7.16/envs/issue/lib/python2.7/site-packages/setuptools/sandbox.py", line 45, in _execfile
        exec(code, globals, locals)
      File "/var/folders/_b/q30qg_l50b5gvqd8y4_wb9h00000gn/T/easy_install-o9MJ5E/numpy-1.17.1/setup.py", line 31, in <module>
        if sys.version_info[0] < 3:
    RuntimeError: Python version >= 3.5 required.
    ----------------------------------------
ERROR: Command errored out with exit status 1: python setup.py egg_info Check the logs for full command output.

About Check the logs for full command output, where to find the logs?

[1fish2]

Good question. Let me suggest:

• If an error message could explain that the explicit command pip install numpy==1.14.6 scipy==1.0.1 is trying to install numpy 1.17.1 when it prints Python version >= 3.5 required, that’d help a lot. (Now that you explained it, I see it implied by last line of the stack trace, but it’s deeply buried.)
• In place of Check the logs for full command output, pip could print something like Rerun with -v for details.
• I’ll post something on Stack Overflow so people searching on this error message can find a clue.

Thanks!

[rgommers]

nice catch, thanks @pradyunsg! easy_install being invoked under the hood explains a few other issues like this we’ve been having.