The Linux wheels broke our build process

We are running Ubuntu 12.04 and 16.04 on amd64 and we’re building several Debian packages that contain NumPy in a Python 2.7 virtualenv. Recently, several of our build processes started failing because of the addition of Linux wheels to NumPy.

The wheels contain pre-built libraries (libgfortran-ed201abd.so.3.0.0 and libopenblasp-r0-39a31c03.2.18.so). For whatever reason, the Ubuntu strip command (used to remove debugging information) fails when called for these two libraries:

strip --remove-section=.comment --remove-section=.note --strip-unneeded libopenblasp-r0-39a31c03.2.18.so
BFD: stRpMnp0: Not enough room for program headers, try linking with -N
strip:stRpMnp0[.note.gnu.build-id]: Bad value
BFD: stRpMnp0: Not enough room for program headers, try linking with -N
strip:stRpMnp0: Bad value

Since strip is automatically called by dh_strip during package build, that lead to the build errors. Even disabling stripping for this binary package lead to errors as dh_shlibdeps (which generates Debian package dependencies based on ELF executable/shlib dependencies) also fails for these binaries.

Steps to reproduce

In a freshly installed Ubuntu 12.04 or Ubuntu 16.04 on amd64, e.g. inside Vagrant:

sudo apt-get install python-virtualenv
python -m virtualenv venv
source venv/bin/activate
pip install -U pip          # pip-8.1.1.tar.gz
pip install -U setuptools   # setuptools-20.10.1
pip install numpy           # numpy-1.11.0-cp27-cp27mu-manylinux1_x86_64.whl
strip --remove-section=.comment --remove-section=.note --strip-unneeded venv/lib/python2.7/site-packages/numpy/.libs/libopenblasp-r0-39a31c03.2.18.so

Possible solutions

• We can (and will) try to install numpy with pip’s --no-binary option in the future
• Remove the wheels from older NumPy releases. The wheels were added back to 1.6.0, which was released in 2011. In our package builds we previously pinned a 6-month-old NumPy version (1.9.3), exactly to avoid such surprises. If the wheels were only added starting from 1.11.0 going forward, this wouldn’t be an issue.

Fixing the strip problem alone is not a viable solution, since dh_shlibdeps still fails after that, because the two libraries are not native to Ubuntu.

Broader issues

The pre-built wheel is great for individual users, but it’s the wrong thing to use for anyone building RPM/Debian packages for others. Even if our build continued to work, we still wouldn’t want to ship generic pre-built libraries. Silently slipping this change into older releases, is going to surprise a lot of people.

It’s still going to surprise people if you start doing this in new releases only, but you could communicate this change more vocally via release notes, etc. so distributors/packagers are aware of it. Ultimately, they should all move to installing NumPy withpip install --no-binary numpy numpy, but that’s not going to happen over night.

[semitom]

Can confirm the problem regarding stripping scipy shared object files with numpy 1.14.0 and scipy 1.0.0.

scipy/special/_ufuncs.cpython-36m-x86_64-linux-gnu.so: ELF load command address/offset not properly aligned

[pv]

Not sure if it’s worth it — moreover, it will invalidate any checksums listed in release notes (although I guess the build tag changes file names, in which case it would not be such a problem?).