BUG: linalg.cholesky: segmentation fault with large matrix

Describe your issue.

I am trying to use gaussian_kde() to fit a 2D KDE model on a dataset with shape (47000, 2). However, I keep getting segmentation fault on a cluster with 40 CPUs and 300+GB memory. I tried to increase the stack limit using ulimit -s, but it didn’t help.

Reproducing Code Example

import numpy as np
import pandas as pd
from scipy.stats import gaussian_kde

df = pd.read_csv('https://gist.githubusercontent.com/xiaohk/925115eda0c5ed2471dc00be5b3d6046/raw/5adc38dce004303b4c413aa9797c53bd99addfe7/test-data-2d.csv')
data = np.stack([df['x'], df['y']], axis=1)
kde_model = gaussian_kde(data, bw_method='silverman')

Error message

Segmentation fault

==== Below is the gdb result ====

Thread 1 "python" received signal SIGSEGV, Segmentation fault.
__memset_avx2_erms () at ../sysdeps/x86_64/multiarch/memset-vec-unaligned-erms.S:145
145     ../sysdeps/x86_64/multiarch/memset-vec-unaligned-erms.S: No such file or directory.
(gdb) backtrace
#0  __memset_avx2_erms () at ../sysdeps/x86_64/multiarch/memset-vec-unaligned-erms.S:145
#1  0x00007fa0587f276c in f2py_rout.flapack_dpotrf () from /nvmescratch/jay/miniconda3/envs/rapids/lib/python3.9/site-packages/scipy/linalg/_flapack.cpython-39-x86_64-linux-gnu.so
#2  0x0000555555688fa7 in _PyObject_MakeTpCall (tstate=0x55555590ddd0, callable=0x7fa058a3e780, args=<optimized out>, nargs=<optimized out>, keywords=<optimized out>) at /usr/local/src/conda/python-3.9.13/Objects/call.c:191
#3  0x000055555568507f in _PyObject_VectorcallTstate (kwnames=0x7fa058a1b5c0, nargsf=<optimized out>, args=<optimized out>, callable=0x7fa058a3e780, tstate=0x55555590ddd0) at /usr/local/src/conda/python-3.9.13/Include/cpython/abstract.h:116
#4  _PyObject_VectorcallTstate (kwnames=0x7fa058a1b5c0, nargsf=<optimized out>, args=<optimized out>, callable=0x7fa058a3e780, tstate=0x55555590ddd0) at /usr/local/src/conda/python-3.9.13/Include/cpython/abstract.h:103
#5  PyObject_Vectorcall (kwnames=0x7fa058a1b5c0, nargsf=<optimized out>, args=<optimized out>, callable=0x7fa058a3e780) at /usr/local/src/conda/python-3.9.13/Include/cpython/abstract.h:127
#6  call_function (kwnames=0x7fa058a1b5c0, oparg=<optimized out>, pp_stack=<synthetic pointer>, tstate=<optimized out>) at /usr/local/src/conda/python-3.9.13/Python/ceval.c:5077
#7  _PyEval_EvalFrameDefault (tstate=<optimized out>, f=<optimized out>, throwflag=<optimized out>) at /usr/local/src/conda/python-3.9.13/Python/ceval.c:3537
#8  0x000055555567ea17 in _PyEval_EvalFrame (throwflag=0, f=0x7ffae9624640, tstate=0x55555590ddd0) at /usr/local/src/conda/python-3.9.13/Include/internal/pycore_ceval.h:40
#9  _PyEval_EvalCode (tstate=<optimized out>, _co=0x7fa0586e22f0, globals=<optimized out>, locals=<optimized out>, args=<optimized out>, argcount=<optimized out>, kwnames=<optimized out>, kwargs=<optimized out>, kwcount=<optimized out>,
    kwstep=<optimized out>, defs=<optimized out>, defcount=<optimized out>, kwdefs=<optimized out>, closure=<optimized out>, name=<optimized out>, qualname=<optimized out>) at /usr/local/src/conda/python-3.9.13/Python/ceval.c:4329
#10 0x0000555555690ff9 in _PyFunction_Vectorcall (func=<optimized out>, stack=<optimized out>, nargsf=<optimized out>, kwnames=<optimized out>) at /usr/local/src/conda/python-3.9.13/Objects/call.c:396
#11 0x0000555555680bcb in _PyObject_VectorcallTstate (kwnames=0x7fa0586e0a90, nargsf=<optimized out>, args=<optimized out>, callable=0x7fa0586ddf70, tstate=0x55555590ddd0) at /usr/local/src/conda/python-3.9.13/Include/cpython/abstract.h:118
#12 PyObject_Vectorcall (kwnames=0x7fa0586e0a90, nargsf=<optimized out>, args=<optimized out>, callable=0x7fa0586ddf70) at /usr/local/src/conda/python-3.9.13/Include/cpython/abstract.h:127
#13 call_function (kwnames=0x7fa0586e0a90, oparg=<optimized out>, pp_stack=<synthetic pointer>, tstate=<optimized out>) at /usr/local/src/conda/python-3.9.13/Python/ceval.c:5077
#14 _PyEval_EvalFrameDefault (tstate=<optimized out>, f=<optimized out>, throwflag=<optimized out>) at /usr/local/src/conda/python-3.9.13/Python/ceval.c:3537
#15 0x000055555567ea17 in _PyEval_EvalFrame (throwflag=0, f=0x7ffabd1f0d60, tstate=0x55555590ddd0) at /usr/local/src/conda/python-3.9.13/Include/internal/pycore_ceval.h:40
#16 _PyEval_EvalCode (tstate=<optimized out>, _co=0x7fa0586e23a0, globals=<optimized out>, locals=<optimized out>, args=<optimized out>, argcount=<optimized out>, kwnames=<optimized out>, kwargs=<optimized out>, kwcount=<optimized out>,
    kwstep=<optimized out>, defs=<optimized out>, defcount=<optimized out>, kwdefs=<optimized out>, closure=<optimized out>, name=<optimized out>, qualname=<optimized out>) at /usr/local/src/conda/python-3.9.13/Python/ceval.c:4329
#17 0x0000555555690ff9 in _PyFunction_Vectorcall (func=<optimized out>, stack=<optimized out>, nargsf=<optimized out>, kwnames=<optimized out>) at /usr/local/src/conda/python-3.9.13/Objects/call.c:396
--Type <RET> for more, q to quit, c to continue without paging--
#18 0x0000555555684636 in _PyObject_VectorcallTstate (kwnames=0x0 <numba.dynamic.globals.7ffd46043b50>, nargsf=<optimized out>, args=0x55555c4f0868, callable=0x7fa0586e6040, tstate=0x55555590ddd0) at /usr/local/src/conda/python-3.9.13/Include/cpython/abstract.h:118
#19 PyObject_Vectorcall (kwnames=0x0 <numba.dynamic.globals.7ffd46043b50>, nargsf=<optimized out>, args=0x55555c4f0868, callable=0x7fa0586e6040) at /usr/local/src/conda/python-3.9.13/Include/cpython/abstract.h:127
#20 call_function (kwnames=0x0 <numba.dynamic.globals.7ffd46043b50>, oparg=<optimized out>, pp_stack=<synthetic pointer>, tstate=0x55555590ddd0) at /usr/local/src/conda/python-3.9.13/Python/ceval.c:5077
#21 _PyEval_EvalFrameDefault (tstate=<optimized out>, f=<optimized out>, throwflag=<optimized out>) at /usr/local/src/conda/python-3.9.13/Python/ceval.c:3489
#22 0x0000555555691303 in _PyEval_EvalFrame (throwflag=0, f=0x55555c4f06e0, tstate=0x55555590ddd0) at /usr/local/src/conda/python-3.9.13/Include/internal/pycore_ceval.h:40
#23 function_code_fastcall (tstate=0x55555590ddd0, co=<optimized out>, args=<optimized out>, nargs=<optimized out>, globals=0x7fa0566b2100) at /usr/local/src/conda/python-3.9.13/Objects/call.c:330
#24 0x0000555555680063 in _PyObject_VectorcallTstate (kwnames=0x0 <numba.dynamic.globals.7ffd46043b50>, nargsf=<optimized out>, args=0x55555c350548, callable=0x7fa0566bd0d0, tstate=0x55555590ddd0) at /usr/local/src/conda/python-3.9.13/Include/cpython/abstract.h:118
#25 PyObject_Vectorcall (kwnames=0x0 <numba.dynamic.globals.7ffd46043b50>, nargsf=<optimized out>, args=0x55555c350548, callable=0x7fa0566bd0d0) at /usr/local/src/conda/python-3.9.13/Include/cpython/abstract.h:127
#26 call_function (kwnames=0x0 <numba.dynamic.globals.7ffd46043b50>, oparg=<optimized out>, pp_stack=<synthetic pointer>, tstate=0x55555590ddd0) at /usr/local/src/conda/python-3.9.13/Python/ceval.c:5077
#27 _PyEval_EvalFrameDefault (tstate=<optimized out>, f=<optimized out>, throwflag=<optimized out>) at /usr/local/src/conda/python-3.9.13/Python/ceval.c:3506
#28 0x000055555567ea17 in _PyEval_EvalFrame (throwflag=0, f=0x55555c3503b0, tstate=0x55555590ddd0) at /usr/local/src/conda/python-3.9.13/Include/internal/pycore_ceval.h:40
#29 _PyEval_EvalCode (tstate=<optimized out>, _co=0x7fa0566bc240, globals=<optimized out>, locals=<optimized out>, args=<optimized out>, argcount=<optimized out>, kwnames=<optimized out>, kwargs=<optimized out>, kwcount=<optimized out>, kwstep=<optimized out>,
    defs=<optimized out>, defcount=<optimized out>, kwdefs=<optimized out>, closure=<optimized out>, name=<optimized out>, qualname=<optimized out>) at /usr/local/src/conda/python-3.9.13/Python/ceval.c:4329
#30 0x00005555556a0328 in _PyFunction_Vectorcall (kwnames=<optimized out>, nargsf=<optimized out>, stack=0x5555576d29f8, func=0x7fa0566bd040) at /usr/local/src/conda/python-3.9.13/Objects/call.c:396
#31 _PyObject_VectorcallTstate (kwnames=<optimized out>, nargsf=<optimized out>, args=0x5555576d29f8, callable=0x7fa0566bd040, tstate=0x55555590ddd0) at /usr/local/src/conda/python-3.9.13/Include/cpython/abstract.h:118
#32 method_vectorcall (method=<optimized out>, args=0x5555576d2a00, nargsf=<optimized out>, kwnames=<optimized out>) at /usr/local/src/conda/python-3.9.13/Objects/classobject.c:53
#33 0x0000555555680bcb in _PyObject_VectorcallTstate (kwnames=0x7fa05673d280, nargsf=<optimized out>, args=<optimized out>, callable=0x7ffd4606ddc0, tstate=0x55555590ddd0) at /usr/local/src/conda/python-3.9.13/Include/cpython/abstract.h:118
#34 PyObject_Vectorcall (kwnames=0x7fa05673d280, nargsf=<optimized out>, args=<optimized out>, callable=0x7ffd4606ddc0) at /usr/local/src/conda/python-3.9.13/Include/cpython/abstract.h:127
#35 call_function (kwnames=0x7fa05673d280, oparg=<optimized out>, pp_stack=<synthetic pointer>, tstate=<optimized out>) at /usr/local/src/conda/python-3.9.13/Python/ceval.c:5077
#36 _PyEval_EvalFrameDefault (tstate=<optimized out>, f=<optimized out>, throwflag=<optimized out>) at /usr/local/src/conda/python-3.9.13/Python/ceval.c:3537
#37 0x000055555567ea17 in _PyEval_EvalFrame (throwflag=0, f=0x5555576d2870, tstate=0x55555590ddd0) at /usr/local/src/conda/python-3.9.13/Include/internal/pycore_ceval.h:40
#38 _PyEval_EvalCode (tstate=<optimized out>, _co=0x7fa0566b7500, globals=<optimized out>, locals=<optimized out>, args=<optimized out>, argcount=<optimized out>, kwnames=<optimized out>, kwargs=<optimized out>, kwcount=<optimized out>, kwstep=<optimized out>,
    defs=<optimized out>, defcount=<optimized out>, kwdefs=<optimized out>, closure=<optimized out>, name=<optimized out>, qualname=<optimized out>) at /usr/local/src/conda/python-3.9.13/Python/ceval.c:4329
#39 0x00005555556886e0 in _PyFunction_Vectorcall (kwnames=0x0 <numba.dynamic.globals.7ffd46043b50>, nargsf=<optimized out>, stack=<optimized out>, func=0x7fa0566b5af0) at /usr/local/src/conda/python-3.9.13/Objects/call.c:396
#40 _PyObject_FastCallDictTstate (tstate=0x55555590ddd0, callable=0x7fa0566b5af0, args=<optimized out>, nargsf=<optimized out>, kwargs=<optimized out>) at /usr/local/src/conda/python-3.9.13/Objects/call.c:118
#41 0x000055555569ced8 in _PyObject_Call_Prepend (kwargs=0x0 <numba.dynamic.globals.7ffd46043b50>, args=0x7ffae80c9ee0, obj=<optimized out>, callable=0x7fa0566b5af0, tstate=0x55555590ddd0) at /usr/local/src/conda/python-3.9.13/Objects/call.c:489
#42 slot_tp_init (self=<optimized out>, args=<optimized out>, kwds=0x0 <numba.dynamic.globals.7ffd46043b50>) at /usr/local/src/conda/python-3.9.13/Objects/typeobject.c:6971
#43 0x0000555555688fbe in type_call (kwds=<optimized out>, args=0x7ffae80c9ee0, type=<optimized out>) at /usr/local/src/conda/python-3.9.13/Objects/typeobject.c:1028
#44 _PyObject_MakeTpCall (tstate=0x55555590ddd0, callable=0x5555564f61e0, args=<optimized out>, nargs=<optimized out>, keywords=<optimized out>) at /usr/local/src/conda/python-3.9.13/Objects/call.c:191
#45 0x00005555556844ae in _PyObject_VectorcallTstate (kwnames=0x0 <numba.dynamic.globals.7ffd46043b50>, nargsf=<optimized out>, args=0x55555596b4a0, callable=<optimized out>, tstate=<optimized out>)
    at /usr/local/src/conda/python-3.9.13/Include/cpython/abstract.h:116
#46 _PyObject_VectorcallTstate (kwnames=0x0 <numba.dynamic.globals.7ffd46043b50>, nargsf=<optimized out>, args=0x55555596b4a0, callable=<optimized out>, tstate=<optimized out>) at /usr/local/src/conda/python-3.9.13/Include/cpython/abstract.h:103
#47 PyObject_Vectorcall (kwnames=0x0 <numba.dynamic.globals.7ffd46043b50>, nargsf=<optimized out>, args=0x55555596b4a0, callable=<optimized out>) at /usr/local/src/conda/python-3.9.13/Include/cpython/abstract.h:127
#48 call_function (kwnames=0x0 <numba.dynamic.globals.7ffd46043b50>, oparg=<optimized out>, pp_stack=<synthetic pointer>, tstate=0x55555590ddd0) at /usr/local/src/conda/python-3.9.13/Python/ceval.c:5077
#49 _PyEval_EvalFrameDefault (tstate=<optimized out>, f=<optimized out>, throwflag=<optimized out>) at /usr/local/src/conda/python-3.9.13/Python/ceval.c:3520
#50 0x000055555567ea17 in _PyEval_EvalFrame (throwflag=0, f=0x55555596b330, tstate=0x55555590ddd0) at /usr/local/src/conda/python-3.9.13/Include/internal/pycore_ceval.h:40
#51 _PyEval_EvalCode (tstate=<optimized out>, _co=0x7ffda3a04870, globals=<optimized out>, locals=<optimized out>, args=<optimized out>, argcount=<optimized out>, kwnames=<optimized out>, kwargs=<optimized out>, kwcount=<optimized out>, kwstep=<optimized out>,
    defs=<optimized out>, defcount=<optimized out>, kwdefs=<optimized out>, closure=<optimized out>, name=<optimized out>, qualname=<optimized out>) at /usr/local/src/conda/python-3.9.13/Python/ceval.c:4329
#52 0x000055555567e6d7 in _PyEval_EvalCodeWithName (_co=<optimized out>, globals=<optimized out>, locals=<optimized out>, args=<optimized out>, argcount=<optimized out>, kwnames=<optimized out>, kwargs=0x0 <numba.dynamic.globals.7ffd46043b50>, kwcount=0, kwstep=2,
    defs=0x0 <numba.dynamic.globals.7ffd46043b50>, defcount=0, kwdefs=0x0 <numba.dynamic.globals.7ffd46043b50>, closure=0x0 <numba.dynamic.globals.7ffd46043b50>, name=0x0 <numba.dynamic.globals.7ffd46043b50>, qualname=0x0 <numba.dynamic.globals.7ffd46043b50>)
    at /usr/local/src/conda/python-3.9.13/Python/ceval.c:4361
#53 0x000055555567e689 in PyEval_EvalCodeEx (_co=<optimized out>, globals=<optimized out>, locals=<optimized out>, args=<optimized out>, argcount=<optimized out>, kws=<optimized out>, kwcount=0, defs=0x0 <numba.dynamic.globals.7ffd46043b50>, defcount=0,
    kwdefs=0x0 <numba.dynamic.globals.7ffd46043b50>, closure=0x0 <numba.dynamic.globals.7ffd46043b50>) at /usr/local/src/conda/python-3.9.13/Python/ceval.c:4377
#54 0x0000555555739e3b in PyEval_EvalCode (co=co@entry=0x7ffda3a04870, globals=globals@entry=0x7ffda39f8d80, locals=locals@entry=0x7ffda39f8d80) at /usr/local/src/conda/python-3.9.13/Python/ceval.c:828
#55 0x00005555557684a9 in run_eval_code_obj (tstate=0x55555590ddd0, co=0x7ffda3a04870, globals=0x7ffda39f8d80, locals=0x7ffda39f8d80) at /usr/local/src/conda/python-3.9.13/Python/pythonrun.c:1221
#56 0x0000555555764694 in run_mod (mod=<optimized out>, filename=<optimized out>, globals=0x7ffda39f8d80, locals=0x7ffda39f8d80, flags=<optimized out>, arena=<optimized out>) at /usr/local/src/conda/python-3.9.13/Python/pythonrun.c:1242
#57 0x00005555555e96d2 in pyrun_file (fp=0x5555559735e0, filename=0x7ffda3981b70, start=<optimized out>, globals=0x7ffda39f8d80, locals=0x7ffda39f8d80, closeit=1, flags=0x7fffffffd568) at /usr/local/src/conda/python-3.9.13/Python/pythonrun.c:1140
#58 0x000055555575e1f2 in pyrun_simple_file (flags=0x7fffffffd568, closeit=1, filename=0x7ffda3981b70, fp=0x5555559735e0) at /usr/local/src/conda/python-3.9.13/Python/pythonrun.c:450
#59 PyRun_SimpleFileExFlags (fp=0x5555559735e0, filename=<optimized out>, closeit=1, flags=0x7fffffffd568) at /usr/local/src/conda/python-3.9.13/Python/pythonrun.c:483
--Type <RET> for more, q to quit, c to continue without paging--
#60 0x000055555575b533 in pymain_run_file (cf=0x7fffffffd568, config=0x55555590e760) at /usr/local/src/conda/python-3.9.13/Modules/main.c:377
#61 pymain_run_python (exitcode=0x7fffffffd560) at /usr/local/src/conda/python-3.9.13/Modules/main.c:602
#62 Py_RunMain () at /usr/local/src/conda/python-3.9.13/Modules/main.c:681
#63 0x000055555572db79 in Py_BytesMain (argc=<optimized out>, argv=<optimized out>) at /usr/local/src/conda/python-3.9.13/Modules/main.c:1101
#64 0x00007ffda293fc87 in __libc_start_main (main=0x55555572db30 <main>, argc=2, argv=0x7fffffffd768, init=<optimized out>, fini=<optimized out>, rtld_fini=<optimized out>, stack_end=0x7fffffffd758) at ../csu/libc-start.c:310
#65 0x000055555572da81 in ?? () at /usr/local/src/conda/python-3.9.13/Python/getplatform.c:9
#66 0x00007fffffffd758 in ?? ()
#67 0x000000000000001c in ?? ()
#68 0x0000000000000002 in ?? ()
#69 0x00007fffffffdaab in ?? ()
#70 0x00007fffffffdade in ?? ()
#71 0x0000000000000000 in ?? ()

SciPy/NumPy/Python version information

1.9.3 1.23.4 sys.version_info(major=3, minor=9, micro=13, releaselevel=‘final’, serial=0)

[xiaohk]

@mdhaber

The segfault happens on this line:

_data_cho_cov = linalg.cholesky(_data_covariance, lower=True)

Yes, after I change data to np.random.rand(47000, 2), the program still crashes with a segfault after running for 25 minutes.

[mdhaber]

Thanks. That import issue has been reported in https://github.com/scipy/scipy/issues/17368.