error during install of dependencies via pip; Python 3.8; gcc version 10.1.0

Hello, I have a problem during the installation of emukit via pip.

I am using Python 3.8.3. I try to install the current version of emukit (0.4.7).

I’ve have tried to install it multiple times, if you want to see the details or you need more information to my setup I’ve created a more detailed gist. I have reproduced the bug as a sort of summary.

How to reproduce the bug

What I wanted to do: I tried to install emukit via pip with the command pip install emukit that is suggested by the README and the Docs

First I create a virtual environment and activate it (I use venv because the first install attempt did not work out of the box, so I did not want to mess up my system and the python packaging tutorial suggested to do so)

python -m venv ./venv
source venv/bin/activate

Then I upgrade pip and install wheel (because the output of pip in previous attempts suggested to do so)

pip install --upgrade pip
pip install wheel

Then I try to install emukit via pip. For the bug report I logged stdout stderr.

pip install emukit 1>install_f.txt  2>error_install_f.txt

The files containing the full output are attached. error_install_f.txt install_f.txt

The error occurs when pip is trying to install scipy==1.1.0 (the fact that pip is trying to install exactly this version of scipy even if a newer is already installed even though emukit doesn’t seem to depend on exactly this version I find odd). The whole error is a very long.

Error: Rank mismatch between actual argument at (1) and actual argument at (2) (scalar and rank-1)
  21     scipy/sparse/linalg/eigen/arpack/ARPACK/SRC/znaitr.f:737:39:

Errors similar to the above line led me to assume it could be related to scipy/scipy#11611

I have also tried to install numpy, gpy and scipy before installing emukit. For details see gist.

Version of python, pip and gcc

$ python -V
Python 3.8.3

$ gcc -v
Using built-in specs.
COLLECT_GCC=gcc
COLLECT_LTO_WRAPPER=/usr/lib/gcc/x86_64-pc-linux-gnu/10.1.0/lto-wrapper
Target: x86_64-pc-linux-gnu
Configured with: /build/gcc/src/gcc/configure --prefix=/usr --libdir=/usr/lib --libexecdir=/usr/lib --mandir=/usr/share/man --infodir=/usr/share/info --with-bugurl=https://bugs.archlinux.org/ --enable-languages=c,c++,ada,fortran,go,lto,objc,obj-c++,d --with-isl --with-linker-hash-style=gnu --with-system-zlib --enable-__cxa_atexit --enable-cet=auto --enable-checking=release --enable-clocale=gnu --enable-default-pie --enable-default-ssp --enable-gnu-indirect-function --enable-gnu-unique-object --enable-install-libiberty --enable-linker-build-id --enable-lto --enable-multilib --enable-plugin --enable-shared --enable-threads=posix --disable-libssp --disable-libstdcxx-pch --disable-libunwind-exceptions --disable-werror gdc_include_dir=/usr/include/dlang/gdc
Thread model: posix
Supported LTO compression algorithms: zlib zstd
gcc version 10.1.0 (GCC) 

$ pip -V   
pip 20.1.1 from /home/<username>/Documents/uni/bayesian-quadrature/venv/lib/python3.8/site-packages/pip (python 3.8)

$ pip list                     
Package    Version
---------- -------
numpy      1.18.5
pip        20.1.1
setuptools 41.2.0
wheel      0.34.2

I appreciate any and all help.

[MashaNaslidnyk]

The issue is with there not being a pre-built wheel for scipy v1.1.0. Or pip not picking it.

This fails: pip install numpy scipy==1.1.0

This succeeds:

pip install numpy scipy==1.4.0

If we insist on installing the earlier version, we need to have C libraries needed to build the wheel installed - this worked for my env: yum install lapack-devel.x86_64 blas-devel.x86_64 pip install numpy scipy==1.1.0

Or, potentially, one one of the .whl here might be suitable - and pip is wrong for deciding it isn’t - in which case one can download it manually. I didn’t try that because it wouldn’t be a reasonable fix to suggest.

We can look into this further; changing scipy==1.1.0 to scipy>=1.1.0 in the meantime seems reasonable as it would solve the issue for many.

As to install order: here’s my educated guess at what’s happening. Pip tries to follow the following:

1. Build all wheels (numpy -> scipy -> …) For numpy, no build is needed - a pre-built wheel is available (and was downloaded before, see line “Using cached numpy-1.18.5-cp38-cp38-manylinux1_x86_64.whl”). For scipy, for some reason, none of the pre-built wheels for v1.1.0 will do, so pip downloads the tarball and attempts to build it.

2. Install all wheels (numpy -> scipy -> …) OR run setup.py install if a wheel build failed

We see two errors in the logs:

1. Scipy wheel can’t be built without numpy - and numpy hasn’t been installed yet, because we’re still at step 1! If numpy is installed (I checked this), we fail to build the wheel still because of missing C libraries, in my case, lapack-devel.x86_64 blas-devel.x86_64 .

But that’s okay because there’s still the option of setup.py install . So the installation proceeds to step 2, installs numpy (this is what the somewhat ambiguous “Installing collected packages: numpy […]” line says), attempts setup.py install of scipy because no wheel was built, and…

1. Fails because of missing C libraries.

The only difference for 1.4.0 is that there is a pre-built wheel available.

[apaleyes]

Update: i am going through recent PRs and issues at the moment, will release new version once I am done with that