MPI: KeyboardInterrupt from numpy.core._multiarray_umath import

I’m trying to run an MPI job with 100 processes on a node with 64 cores by using the “oversubscribe” option with mpirun. I can successfully run

 mpirun -np 100 -map-by core -oversubscribe python -c 'print("hello world")' 

but I get an error when I run

mpirun -np 100 -map-by core -oversubscribe python -c 'import numpy as np; print("hello world")' 

The error message is below (64 cores printed at the same time, so these lines may not be in order)

IMPORTANT: PLEASE READ THIS FOR ADVICE ON HOW TO SOLVE THIS ISSUE!

Importing the numpy c-extensions failed.
- Try uninstalling and reinstalling numpy.
- If you have already done that, then:
  1. Check that you expected to use Python3.8 from "/home/james.nance/miniconda3/envs/mpi_test/bin/python",
     and that you have no directories in your PATH or PYTHONPATH that can
     interfere with the Python and numpy version "1.17.3" you're trying to use.
  2. If (1) looks fine, you can open a new issue at
     https://github.com/numpy/numpy/issues.  Please include details on:
     - how you installed Python
     - how you installed numpy
     - your operating system
     - whether or not you have multiple versions of Python installed
     - if you built from source, your compiler versions and ideally a build log

- If you're working with a numpy git repository, try `git clean -xdf`
  (removes all files not under version control) and rebuild numpy.

Note: this error has many possible causes, so please don't comment on
an existing issue about this - open a new one instead.

Original error was: PyCapsule_Import could not import module "datetime"

ly unavailable
OpenBLAS blas_thread_init: RLIMIT_NPROC 4096 current, 1030652 max
OpenBLAS blas_thread_init: pthread_create failed for thread 51 of 64: Resource temporarily unavailable
OpenBLAS blas_thread_init: RLIMIT_NPROC 4096 current, 1030652 max
OpenBLAS blas_thread_init: pthread_create failed for thread 52 of 64: Resource temporarily unavailable
OpenBLAS blas_thread_init: RLIMIT_NPROC 4096 current, 1030652 max
OpenBLAS blas_thread_init: pthread_create failed for thread 53 of 64: Resource temporarily unavailable
OpenBLAS blas_thread_init: RLIMIT_NPROC 4096 current, 1030652 max
OpenBLAS blas_thread_init: pthread_create failed for thread 54 of 64: Resource temporarily unavailable
OpenBLAS blas_thread_init: RLIMIT_NPROC 4096 current, 1030652 max
OpenBLAS blas_thread_init: pthread_create failed for thread 55 of 64: Resource temporarily unavailable
OpenBLAS blas_thread_init: RLIMIT_NPROC 4096 current, 1030652 max
OpenBLAS blas_thread_init: pthread_create failed for thread 56 of 64: Resource temporarily unavailable
OpenBLAS blas_thread_init: RLIMIT_NPROC 4096 current, 1030652 max
OpenBLAS blas_thread_init: pthread_create failed for thread 57 of 64: Resource temporarily unavailable
OpenBLAS blas_thread_init: RLIMIT_NPROC 4096 current, 1030652 max
OpenBLAS blas_thread_init: pthread_create failed for thread 58 of 64: Resource temporarily unavailable
OpenBLAS blas_thread_init: RLIMIT_NPROC 4096 current, 1030652 max
OpenBLAS blas_thread_init: pthread_create failed for thread 59 of 64: Resource temporarily unavailable
OpenBLAS blas_thread_init: RLIMIT_NPROC 4096 current, 1030652 max
OpenBLAS blas_thread_init: pthread_create failed for thread 60 of 64: Resource temporarily unavailable
OpenBLAS blas_thread_init: RLIMIT_NPROC 4096 current, 1030652 max
OpenBLAS blas_thread_init: pthread_create failed for thread 61 of 64: Resource temporarily unavailable
OpenBLAS blas_thread_init: RLIMIT_NPROC 4096 current, 1030652 max
OpenBLAS blas_thread_init: pthread_create failed for thread 62 of 64: Resource temporarily unavailable
OpenBLAS blas_thread_init: RLIMIT_NPROC 4096 current, 1030652 max
OpenBLAS blas_thread_init: pthread_create failed for thread 63 of 64: Resource temporarily unavailable
OpenBLAS blas_thread_init: RLIMIT_NPROC 4096 current, 1030652 max
Traceback (most recent call last):
  File "/home/james.nance/miniconda3/envs/mpi_test/lib/python3.8/site-packages/numpy/core/__init__.py", line 17, in <module>
    from . import multiarray
  File "/home/james.nance/miniconda3/envs/mpi_test/lib/python3.8/site-packages/numpy/core/multiarray.py", line 14, in <module>
    from . import overrides
  File "/home/james.nance/miniconda3/envs/mpi_test/lib/python3.8/site-packages/numpy/core/overrides.py", line 7, in <module>
    from numpy.core._multiarray_umath import (
ImportError: PyCapsule_Import could not import module "datetime"

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/home/james.nance/miniconda3/envs/mpi_test/lib/python3.8/site-packages/numpy/__init__.py", line 142, in <module>
    from . import core
  File "/home/james.nance/miniconda3/envs/mpi_test/lib/python3.8/site-packages/numpy/core/__init__.py", line 47, in <module>
    raise ImportError(msg)
ImportError: 

After some researching, I tried to import datetime before numpy

mpirun -np 100 -map-by core -oversubscribe python -c 'import datetime; import numpy as np; print("hello world")' 

and get this error message:

Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/home/james.nance/miniconda3/envs/mpi_test/lib/python3.8/site-packages/numpy/__init__.py", line 142, in <module>
    from . import core
  File "/home/james.nance/miniconda3/envs/mpi_test/lib/python3.8/site-packages/numpy/core/__init__.py", line 17, in <module>
    from . import multiarray
  File "/home/james.nance/miniconda3/envs/mpi_test/lib/python3.8/site-packages/numpy/core/multiarray.py", line 14, in <module>
    from . import overrides
  File "/home/james.nance/miniconda3/envs/mpi_test/lib/python3.8/site-packages/numpy/core/overrides.py", line 7, in <module>
    from numpy.core._multiarray_umath import (
KeyboardInterrupt
OpenBLAS blas_thread_init: pthread_create failed for thread 21 of 64: Resource temporarily unavailable
OpenBLAS blas_thread_init: RLIMIT_NPROC 4096 current, 1030652 max
OpenBLAS blas_thread_init: pthread_create failed for thread 22 of 64: Resource temporarily unavailable
OpenBLAS blas_thread_init: RLIMIT_NPROC 4096 current, 1030652 max
OpenBLAS blas_thread_init: pthread_create failed for thread 23 of 64: Resource temporarily unavailable
OpenBLAS blas_thread_init: RLIMIT_NPROC 4096 current, 1030652 max
OpenBLAS blas_thread_init: pthread_create failed for thread 24 of 64: Resource temporarily unavailable

Any help would be appreciated!

Version/OS info: Numpy version 1.17.3 (installed via conda) Python version 3.8.0 (installed via conda) Operating System: CentOS Linux 7 (Core) Kernel: Linux 3.10.0-957.27.2.el7.x86_64 Architecture: x86-64

[wonderboy3489]

import os;
os.environ["OMP_NUM_THREADS"] = "1"

before import numpy seems to do the trick. Thanks for all of your suggestions!

[seberg]

Sounds like it was an issue of oversubscribing and setting openblas to use a single thread fixes it. There are other issues about the more general problem open I believe (and i tis something numpy has little direct control over).

So closing.