RuntimeError: module compiled against API version 0x10 but this version of numpy is 0xe

Sorry for the repetition. I have trained a speech enhancement task in the following environment. However, an error occurs and it stops in the middle of the first epoch.

The environment is as follows

PyTorch version: 1.10.1
Is debug build: False
CUDA used to build PyTorch: 11.3
ROCM used to build PyTorch: N/A

OS: Ubuntu 20.04.4 LTS (x86_64)
GCC version: (Ubuntu 9.4.0-1ubuntu1~20.04.1) 9.4.0
Clang version: Could not collect
CMake version: version 3.16.3
Libc version: glibc-2.31

Python version: 3.8.13 (default, Mar 28 2022, 11:38:47)  [GCC 7.5.0] (64-bit runtime)
Python platform: Linux-5.10.16.3-microsoft-standard-WSL2-x86_64-with-glibc2.17
Is CUDA available: True
CUDA runtime version: 11.3.109
GPU models and configuration: GPU 0: NVIDIA GeForce RTX 3060
Nvidia driver version: 516.93
cuDNN version: Could not collect
HIP runtime version: N/A
MIOpen runtime version: N/A

Versions of relevant libraries:
[pip3] numpy==1.21.5
[pip3] pytorch-ranger==0.1.1
[pip3] pytorch-wpe==0.0.1
[pip3] torch==1.10.1
[pip3] torch-complex==0.4.3
[pip3] torch-optimizer==0.3.0
[pip3] torchaudio==0.10.1
[conda] blas                      1.0                         mkl
[conda] cudatoolkit               11.3.1               h2bc3f7f_2
[conda] mkl                       2021.4.0           h06a4308_640
[conda] mkl-service               2.4.0            py38h7f8727e_0
[conda] mkl_fft                   1.3.1            py38hd3c417c_0
[conda] mkl_random                1.2.2            py38h51133e4_0
[conda] numpy                     1.21.4                   pypi_0    pypi
[conda] numpy-base                1.21.5           py38ha15fc14_3
[conda] pytorch                   1.10.1          py3.8_cuda11.3_cudnn8.2.0_0    pytorch
[conda] pytorch-mutex             1.0                        cuda    pytorch
[conda] pytorch-ranger            0.1.1                    pypi_0    pypi
[conda] pytorch-wpe               0.0.1                    pypi_0    pypi
[conda] torch-complex             0.4.3                    pypi_0    pypi
[conda] torch-optimizer           0.3.0                    pypi_0    pypi
[conda] torchaudio                0.10.1               py38_cu113    pytorch

train.log

I had no problem with Python 3.7.

How should I deal with this? Best regards.

[kamo-naoyuki]

FYI: To reproduce your error

pip install -U numpy
pip uninstall matplotlib
pip install --no-cache-dir "matplotlib==3.1.0"  # matplotlib was compiled with latest numpy
pip install numpy==1.21.4
python -c "import matplotlib.pyplot"

To fix this problem

pip uninstall matplotlib
pip install --no-cache-dir "matplotlib==3.1.0"  # Recompile matplotlib with the current numpy

Note that Numpy-ABI has forward compatibility, but backward compatibility is not guaranteed.

(I don’t know how you conda environment was created yet)

[kamo-naoyuki]

While[conda] numpy describes the package managed by conda, [pip3] numpy is the package installed by pip3. Note that conda originally is aware of pip, thus this should normally be the same.

I don’t know how it happens, but, probably, the pip3 is not managed by conda in your environment (e.g. You overwrited the path of python/pip)

Check your pip

which pip3

Or, check your import path, one by one

python -c "import sys; print(sys.path)"

This is not an issue of espnet, but a problem of your environment. I’ll close this issue.