Double free or corruption on numba import w/ icc-rt

Hey Stuart and Siu,

I began having build errors for numba-accelerated libraries yesterday. I think it may be related to icc-rt. For example:

no-icc-rt env (this works fine):

(no-icc-rt) root@b9fa89e79c6b:/workspaces/numba-stats# conda list
# packages in environment at /opt/conda/envs/no-icc-rt:
#
# Name                    Version                   Build  Channel
_libgcc_mutex             0.1                 conda_forge    conda-forge
_openmp_mutex             4.5                       1_gnu    conda-forge
ca-certificates           2020.11.8            ha878542_0    conda-forge
certifi                   2020.11.8        py37h89c1867_0    conda-forge
ld_impl_linux-64          2.35                 h769bd43_9    conda-forge
libblas                   3.9.0                2_openblas    conda-forge
libcblas                  3.9.0                2_openblas    conda-forge
libffi                    3.2.1             he1b5a44_1007    conda-forge
libgcc-ng                 9.3.0               h5dbcf3e_17    conda-forge
libgfortran-ng            9.3.0               he4bcb1c_17    conda-forge
libgfortran5              9.3.0               he4bcb1c_17    conda-forge
libgomp                   9.3.0               h5dbcf3e_17    conda-forge
liblapack                 3.9.0                2_openblas    conda-forge
libllvm10                 10.0.1               he513fc3_3    conda-forge
libopenblas               0.3.12          pthreads_h4812303_1    conda-forge
libstdcxx-ng              9.3.0               h2ae2ef3_17    conda-forge
llvmlite                  0.34.0           py37h5202443_2    conda-forge
ncurses                   6.2                  h58526e2_3    conda-forge
numba                     0.51.2           py37h9fdb41a_0    conda-forge
numpy                     1.19.4           py37h7e9df27_1    conda-forge
openssl                   1.1.1h               h516909a_0    conda-forge
pip                       20.2.4                     py_0    conda-forge
python                    3.7.8           h6f2ec95_1_cpython    conda-forge
python_abi                3.7                     1_cp37m    conda-forge
readline                  8.0                  he28a2e2_2    conda-forge
setuptools                49.6.0           py37he5f6b98_2    conda-forge
sqlite                    3.33.0               h4cf870e_1    conda-forge
tk                        8.6.10               hed695b0_1    conda-forge
wheel                     0.35.1             pyh9f0ad1d_0    conda-forge
xz                        5.2.5                h516909a_1    conda-forge
zlib                      1.2.11            h516909a_1010    conda-forge
(no-icc-rt) root@b9fa89e79c6b:/workspaces/numba-stats# python
Python 3.7.8 | packaged by conda-forge | (default, Jul 31 2020, 02:25:08) 
[GCC 7.5.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import numba as nb
>>> exit()

yes-icc-rt env (abort due to corruption):

(yes-icc-rt) root@b9fa89e79c6b:/workspaces/numba-stats# conda list
# packages in environment at /opt/conda/envs/yes-icc-rt:
#
# Name                    Version                   Build  Channel
_libgcc_mutex             0.1                 conda_forge    conda-forge
_openmp_mutex             4.5                       1_gnu    conda-forge
ca-certificates           2020.11.8            ha878542_0    conda-forge
certifi                   2020.11.8        py37h89c1867_0    conda-forge
icc_rt                    2020.2                intel_254    numba
ld_impl_linux-64          2.35                 h769bd43_9    conda-forge
libblas                   3.9.0                2_openblas    conda-forge
libcblas                  3.9.0                2_openblas    conda-forge
libffi                    3.2.1             he1b5a44_1007    conda-forge
libgcc-ng                 9.3.0               h5dbcf3e_17    conda-forge
libgfortran-ng            9.3.0               he4bcb1c_17    conda-forge
libgfortran5              9.3.0               he4bcb1c_17    conda-forge
libgomp                   9.3.0               h5dbcf3e_17    conda-forge
liblapack                 3.9.0                2_openblas    conda-forge
libllvm10                 10.0.1               he513fc3_3    conda-forge
libopenblas               0.3.12          pthreads_h4812303_1    conda-forge
libstdcxx-ng              9.3.0               h2ae2ef3_17    conda-forge
llvmlite                  0.34.0           py37h5202443_2    conda-forge
ncurses                   6.2                  h58526e2_3    conda-forge
numba                     0.51.2           py37h9fdb41a_0    conda-forge
numpy                     1.19.4           py37h7e9df27_1    conda-forge
openssl                   1.1.1h               h516909a_0    conda-forge
pip                       20.2.4                     py_0    conda-forge
python                    3.7.8           h6f2ec95_1_cpython    conda-forge
python_abi                3.7                     1_cp37m    conda-forge
readline                  8.0                  he28a2e2_2    conda-forge
setuptools                49.6.0           py37he5f6b98_2    conda-forge
sqlite                    3.33.0               h4cf870e_1    conda-forge
tk                        8.6.10               hed695b0_1    conda-forge
wheel                     0.35.1             pyh9f0ad1d_0    conda-forge
xz                        5.2.5                h516909a_1    conda-forge
zlib                      1.2.11            h516909a_1010    conda-forge
(yes-icc-rt) root@b9fa89e79c6b:/workspaces/numba-stats# python
Python 3.7.8 | packaged by conda-forge | (default, Jul 31 2020, 02:25:08) 
[GCC 7.5.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import numba as nb
double free or corruption (top)
Aborted
(yes-icc-rt) root@b9fa89e79c6b:/workspaces/numba-stats# 

Any thoughts would be helpful. Thanks!

This is on an Ubuntu 18.04 on Docker v19.03.13 using Hyper-V.

BR, Ryan

[stuartarchibald]

Also if you don’t mind - hoping to learn something about memory pointers. If ctypes.CDLL(None) points to the following memory address:

>>> ctypes.CDLL(None)
<CDLL 'None', handle 7f22d9b84170 at 0x7f22d87d6f50>

Then why does trying to free 0x1 cause a segfault?

Ref: https://docs.python.org/3/library/ctypes.html#loading-shared-libraries

All these classes can be instantiated by calling them with at least one argument, the pathname of the shared library. If you have an existing handle to an already loaded shared library, it can be passed as the handle named parameter, otherwise the underlying platforms dlopen or LoadLibrary function is used to load the library into the process, and to get a handle to it.

As a result CDLL(None) is asking to open the shared library None, None is synonymous with NULL in this case, which means the call looks like the C code dlopen(NULL);. See dlopen(3) for details, the important bit being:

If filename is NULL, then the returned handle is for the main program.

As a result, lib = ctypes.CDLL(None) puts a handle to the main program into lib, and because Python is linked against libc the symbol free(3) is available, and so free(0x1) is just calling the libc free function with argument 0x1, which is segmentation fault territory…

[mfoglio]

Did you find a solution for this?