ImportError "No module named numpy.core._multiarray_umath" on PPC64le

The error that I’m seeing is this:

error message

$ python
Python 3.6.8 (default, Jun 11 2019, 14:54:15) 
[GCC 4.8.5 20150623 (Red Hat 4.8.5-39)] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import numpy
Traceback (most recent call last):
  File "/home/jenkins/tmpenv/lib/python3.6/site-packages/numpy/core/__init__.py", line 24, in <module>
    from . import multiarray
  File "/home/jenkins/tmpenv/lib/python3.6/site-packages/numpy/core/multiarray.py", line 14, in <module>
    from . import overrides
  File "/home/jenkins/tmpenv/lib/python3.6/site-packages/numpy/core/overrides.py", line 7, in <module>
    from numpy.core._multiarray_umath import (
ModuleNotFoundError: No module named 'numpy.core._multiarray_umath'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/jenkins/tmpenv/lib/python3.6/site-packages/numpy/__init__.py", line 142, in <module>
    from . import core
  File "/home/jenkins/tmpenv/lib/python3.6/site-packages/numpy/core/__init__.py", line 54, in <module>
    raise ImportError(msg)
ImportError: 

IMPORTANT: PLEASE READ THIS FOR ADVICE ON HOW TO SOLVE THIS ISSUE!

Importing the numpy c-extensions failed.
- Try uninstalling and reinstalling numpy.
- If you have already done that, then:
  1. Check that you expected to use Python3.6 from "/home/jenkins/tmpenv/bin/python",
     and that you have no directories in your PATH or PYTHONPATH that can
     interfere with the Python and numpy version "1.18.1" you're trying to use.
  2. If (1) looks fine, you can open a new issue at
     https://github.com/numpy/numpy/issues.  Please include details on:
     - how you installed Python
     - how you installed numpy
     - your operating system
     - whether or not you have multiple versions of Python installed
     - if you built from source, your compiler versions and ideally a build log

- If you're working with a numpy git repository, try `git clean -xdf`
  (removes all files not under version control) and rebuild numpy.

Note: this error has many possible causes, so please don't comment on
an existing issue about this - open a new one instead.

Original error was: No module named 'numpy.core._multiarray_umath'

Since the message says “you can open a new issue …”, I decided why not – I mean, who doesn’t like to deal with obscure installation errors? Some more details about the overall setup:

- how you installed Python To be honest, I didn’t, the python was already pre-installed on the machine. I merely created a virtual environment (virtualenv tmpenv --python=/usr/bin/python3.6 && source tmpenv/bin/activate). The python version is displayed as Python 3.6.8 (default, Jun 11 2019, 14:54:15)

- how you installed numpy

$ pip install numpy -i https://h2oai.github.io/py-repo/
Looking in indexes: https://h2oai.github.io/py-repo/
Collecting numpy
  Downloading http://artifacts.h2o.ai.s3.amazonaws.com/releases/ai/h2o/thirdparty/numpy/numpy-1.18.1-cp36-cp36m-linux_ppc64le.whl (15.4 MB)
     |████████████████████████████████| 15.4 MB 503 kB/s 
Installing collected packages: numpy
Successfully installed numpy-1.18.1

This is a pre-built wheel, which was essentially built using pip wheel numpy inside the official docker image for manylinux2014-powerpc64le. Full details are here: https://github.com/h2oai/py-repo/tree/master/numpy.

- your operating system The OS is Red Hat Enterprise Linux 7.7 on a PowerPC64le machine.

- whether or not you have multiple versions of Python installed I’m working in a virtual environment, so it is equivalent to having only one python.

- if you built from source, your compiler versions and ideally a build log Sure, why not. I’ve added a -vv flag to the script linked above, and here’s the output: numpy-build-log.txt

---

As it happens, I even know why the error occurs. The following files are installed in the numpy directory:

$ ls tmpenv/lib/python3.6/site-packages/numpy/core/*.so
tmpenv/lib/python3.6/site-packages/numpy/core/_multiarray_tests.cpython-36m-powerpc64le-linux-gnu.so
tmpenv/lib/python3.6/site-packages/numpy/core/_multiarray_umath.cpython-36m-powerpc64le-linux-gnu.so
tmpenv/lib/python3.6/site-packages/numpy/core/_operand_flag_tests.cpython-36m-powerpc64le-linux-gnu.so
tmpenv/lib/python3.6/site-packages/numpy/core/_rational_tests.cpython-36m-powerpc64le-linux-gnu.so
tmpenv/lib/python3.6/site-packages/numpy/core/_struct_ufunc_tests.cpython-36m-powerpc64le-linux-gnu.so
tmpenv/lib/python3.6/site-packages/numpy/core/_umath_tests.cpython-36m-powerpc64le-linux-gnu.so

At the same time, the python says the following:

>>> import sysconfig
>>> sysconfig.get_config_var("EXT_SUFFIX")
'.cpython-36m-ppc64le-linux-gnu.so'

As you can see, there is a mismatch between the extension on the files that are installed, and the extension that the python expects: powerpc64le vs ppc64le. Renaming these (and all other .so files in other directories) so that they have “ppc64le” in their name solves the problem: numpy works. However, this is a one-time solution.

What I really don’t understand is whose responsibility it is to fix this problem permanently?

• Is this a faulty python distribution (though upgrading python on a server might be hard)?
• Should it be pip’s responsibility to rename the files during install?
• The numpy itself could notice the mismatch in tags and do something about it?
• Wheel created incorrectly? (however, I tried on a different PPC64 machine, and the same wheel works perfectly fine there. That machine has python 3.6.5)

[mattip]

Let’s keep the conversation on pypa/manylinux since this is not a numpy-specific problem.

[seberg]

Not sure about the status, but it seems like some provider of the cpythong executable is using the wrong extension for things, and not something that NumPy can do about. The pypa issue probably more relevant anyway.