Using numba and julia (or any other LLVM loader?) at the same time leads to segfaults

Reporting a bug

• I have tried using the latest released version of Numba
• I have included a self contained code sample to reproduce the problem.

There are several bugs related to julia (here, here, and here) that relate to segmentation faults when trying to use numba after julia has loaded. The segfaults can be triggered either by calling numba from julia, or by calling numba from python after loading julia. They appear to be due to the fact that julia has already loaded (and presumably used) a different version of libLLVM (currently 12) than numba uses (currently 11).

Julia has implemented symbol versioning in the build of libLLVM that it ships along with julia binaries. So the suggestion from the julia side is that there is likely an ffi call in numba (or maybe llvmlite?) that is missing a library name.

If this is true, then it seems like the bug could be triggered without julia at all, but would require loading (and using?) the different version of libLLVM, which is not something I know how to do. But it also suggests that this would be a problem any time someone wants to use numba alongside anything that uses a different libLLVM. So here’s the simplest reproducer I can use, which does use julia from python (and therefore assumes julia has also been installed):

> mamba create -n test_numba_segfault numba
> conda activate test_numba_segfault
> python -m pip install julia
> python
>>> from julia.api import Julia
>>> jl = Julia(compiled_modules=False)
>>> import numba
>>> numba.njit(lambda a, b: a+b)

The resulting segfault looks like this:

signal (11): Segmentation fault: 11
in expression starting at none:0
_ZN4llvm17PMTopLevelManager16addImmutablePassEPNS_13ImmutablePassE at /Users/me/miniconda3/envs/test_numba_segfault/lib/libLLVM-11.dylib (unknown line)
_ZN4llvm17PMTopLevelManager12schedulePassEPNS_4PassE at /Users/me/miniconda3/envs/test_numba_segfault/lib/libLLVM-11.dylib (unknown line)
_ZN4llvm17PMTopLevelManager12schedulePassEPNS_4PassE at /Users/me/miniconda3/envs/test_numba_segfault/lib/libLLVM-11.dylib (unknown line)
_ZN4llvm18PassManagerBuilder25populateModulePassManagerERNS_6legacy15PassManagerBaseE at /Users/me/miniconda3/envs/test_numba_segfault/lib/libLLVM-11.dylib (unknown line)
ffi_call_unix64 at /Users/me/miniconda3/envs/test_numba_segfault/lib/libffi.8.dylib (unknown line)
ffi_call_int at /Users/me/miniconda3/envs/test_numba_segfault/lib/libffi.8.dylib (unknown line)
_ctypes_callproc at /Users/me/miniconda3/envs/test_numba_segfault/lib/python3.10/lib-dynload/_ctypes.cpython-310-darwin.so (unknown line)
PyCFuncPtr_call at /Users/me/miniconda3/envs/test_numba_segfault/lib/python3.10/lib-dynload/_ctypes.cpython-310-darwin.so (unknown line)
_PyObject_Call at /Users/me/miniconda3/envs/test_numba_segfault/bin/python (unknown line)
_PyEval_EvalFrameDefault at /Users/me/miniconda3/envs/test_numba_segfault/bin/python (unknown line)
_PyEval_Vector at /Users/me/miniconda3/envs/test_numba_segfault/bin/python (unknown line)
_PyObject_FastCallDictTstate at /Users/me/miniconda3/envs/test_numba_segfault/bin/python (unknown line)
slot_tp_call at /Users/me/miniconda3/envs/test_numba_segfault/bin/python (unknown line)
_PyObject_MakeTpCall at /Users/me/miniconda3/envs/test_numba_segfault/bin/python (unknown line)
call_function at /Users/me/miniconda3/envs/test_numba_segfault/bin/python (unknown line)
_PyEval_EvalFrameDefault at /Users/me/miniconda3/envs/test_numba_segfault/bin/python (unknown line)
_PyEval_Vector at /Users/me/miniconda3/envs/test_numba_segfault/bin/python (unknown line)
call_function at /Users/me/miniconda3/envs/test_numba_segfault/bin/python (unknown line)
_PyEval_EvalFrameDefault at /Users/me/miniconda3/envs/test_numba_segfault/bin/python (unknown line)
_PyEval_Vector at /Users/me/miniconda3/envs/test_numba_segfault/bin/python (unknown line)
call_function at /Users/me/miniconda3/envs/test_numba_segfault/bin/python (unknown line)
_PyEval_EvalFrameDefault at /Users/me/miniconda3/envs/test_numba_segfault/bin/python (unknown line)
_PyEval_Vector at /Users/me/miniconda3/envs/test_numba_segfault/bin/python (unknown line)
method_vectorcall at /Users/me/miniconda3/envs/test_numba_segfault/bin/python (unknown line)
call_function at /Users/me/miniconda3/envs/test_numba_segfault/bin/python (unknown line)
_PyEval_EvalFrameDefault at /Users/me/miniconda3/envs/test_numba_segfault/bin/python (unknown line)
_PyEval_Vector at /Users/me/miniconda3/envs/test_numba_segfault/bin/python (unknown line)
call_function at /Users/me/miniconda3/envs/test_numba_segfault/bin/python (unknown line)
_PyEval_EvalFrameDefault at /Users/me/miniconda3/envs/test_numba_segfault/bin/python (unknown line)
_PyEval_Vector at /Users/me/miniconda3/envs/test_numba_segfault/bin/python (unknown line)
_PyObject_FastCallDictTstate at /Users/me/miniconda3/envs/test_numba_segfault/bin/python (unknown line)
slot_tp_init at /Users/me/miniconda3/envs/test_numba_segfault/bin/python (unknown line)
type_call at /Users/me/miniconda3/envs/test_numba_segfault/bin/python (unknown line)
_PyObject_MakeTpCall at /Users/me/miniconda3/envs/test_numba_segfault/bin/python (unknown line)
call_function at /Users/me/miniconda3/envs/test_numba_segfault/bin/python (unknown line)
_PyEval_EvalFrameDefault at /Users/me/miniconda3/envs/test_numba_segfault/bin/python (unknown line)
_PyEval_Vector at /Users/me/miniconda3/envs/test_numba_segfault/bin/python (unknown line)
_PyEval_EvalFrameDefault at /Users/me/miniconda3/envs/test_numba_segfault/bin/python (unknown line)
_PyEval_Vector at /Users/me/miniconda3/envs/test_numba_segfault/bin/python (unknown line)
call_function at /Users/me/miniconda3/envs/test_numba_segfault/bin/python (unknown line)
_PyEval_EvalFrameDefault at /Users/me/miniconda3/envs/test_numba_segfault/bin/python (unknown line)
_PyEval_Vector at /Users/me/miniconda3/envs/test_numba_segfault/bin/python (unknown line)
method_vectorcall at /Users/me/miniconda3/envs/test_numba_segfault/bin/python (unknown line)
call_function at /Users/me/miniconda3/envs/test_numba_segfault/bin/python (unknown line)
_PyEval_EvalFrameDefault at /Users/me/miniconda3/envs/test_numba_segfault/bin/python (unknown line)
_PyEval_Vector at /Users/me/miniconda3/envs/test_numba_segfault/bin/python (unknown line)
_PyObject_FastCallDictTstate at /Users/me/miniconda3/envs/test_numba_segfault/bin/python (unknown line)
slot_tp_init at /Users/me/miniconda3/envs/test_numba_segfault/bin/python (unknown line)
type_call at /Users/me/miniconda3/envs/test_numba_segfault/bin/python (unknown line)
_PyObject_MakeTpCall at /Users/me/miniconda3/envs/test_numba_segfault/bin/python (unknown line)
call_function at /Users/me/miniconda3/envs/test_numba_segfault/bin/python (unknown line)
_PyEval_EvalFrameDefault at /Users/me/miniconda3/envs/test_numba_segfault/bin/python (unknown line)
_PyEval_Vector at /Users/me/miniconda3/envs/test_numba_segfault/bin/python (unknown line)
call_function at /Users/me/miniconda3/envs/test_numba_segfault/bin/python (unknown line)
_PyEval_EvalFrameDefault at /Users/me/miniconda3/envs/test_numba_segfault/bin/python (unknown line)
_PyEval_Vector at /Users/me/miniconda3/envs/test_numba_segfault/bin/python (unknown line)
object_vacall at /Users/me/miniconda3/envs/test_numba_segfault/bin/python (unknown line)
PyObject_CallFunctionObjArgs at /Users/me/miniconda3/envs/test_numba_segfault/bin/python (unknown line)
_PyObject_GenericGetAttrWithDict at /Users/me/miniconda3/envs/test_numba_segfault/bin/python (unknown line)
_PyEval_EvalFrameDefault at /Users/me/miniconda3/envs/test_numba_segfault/bin/python (unknown line)
_PyEval_Vector at /Users/me/miniconda3/envs/test_numba_segfault/bin/python (unknown line)
property_descr_get at /Users/me/miniconda3/envs/test_numba_segfault/bin/python (unknown line)
_PyObject_GenericGetAttrWithDict at /Users/me/miniconda3/envs/test_numba_segfault/bin/python (unknown line)
_PyEval_EvalFrameDefault at /Users/me/miniconda3/envs/test_numba_segfault/bin/python (unknown line)
_PyEval_Vector at /Users/me/miniconda3/envs/test_numba_segfault/bin/python (unknown line)
_PyObject_FastCallDictTstate at /Users/me/miniconda3/envs/test_numba_segfault/bin/python (unknown line)
slot_tp_init at /Users/me/miniconda3/envs/test_numba_segfault/bin/python (unknown line)
type_call at /Users/me/miniconda3/envs/test_numba_segfault/bin/python (unknown line)
_PyObject_Call at /Users/me/miniconda3/envs/test_numba_segfault/bin/python (unknown line)
_PyEval_EvalFrameDefault at /Users/me/miniconda3/envs/test_numba_segfault/bin/python (unknown line)
_PyEval_Vector at /Users/me/miniconda3/envs/test_numba_segfault/bin/python (unknown line)
call_function at /Users/me/miniconda3/envs/test_numba_segfault/bin/python (unknown line)
_PyEval_EvalFrameDefault at /Users/me/miniconda3/envs/test_numba_segfault/bin/python (unknown line)
_PyEval_Vector at /Users/me/miniconda3/envs/test_numba_segfault/bin/python (unknown line)
PyVectorcall_Call at /Users/me/miniconda3/envs/test_numba_segfault/bin/python (unknown line)
_PyEval_EvalFrameDefault at /Users/me/miniconda3/envs/test_numba_segfault/bin/python (unknown line)
_PyEval_Vector at /Users/me/miniconda3/envs/test_numba_segfault/bin/python (unknown line)
call_function at /Users/me/miniconda3/envs/test_numba_segfault/bin/python (unknown line)
_PyEval_EvalFrameDefault at /Users/me/miniconda3/envs/test_numba_segfault/bin/python (unknown line)
_PyEval_Vector at /Users/me/miniconda3/envs/test_numba_segfault/bin/python (unknown line)
PyRun_InteractiveOneObjectEx at /Users/me/miniconda3/envs/test_numba_segfault/bin/python (unknown line)
_PyRun_InteractiveLoopObject at /Users/me/miniconda3/envs/test_numba_segfault/bin/python (unknown line)
_PyRun_AnyFileObject at /Users/me/miniconda3/envs/test_numba_segfault/bin/python (unknown line)
PyRun_AnyFileExFlags at /Users/me/miniconda3/envs/test_numba_segfault/bin/python (unknown line)
pymain_run_stdin at /Users/me/miniconda3/envs/test_numba_segfault/bin/python (unknown line)
pymain_run_python at /Users/me/miniconda3/envs/test_numba_segfault/bin/python (unknown line)
Py_RunMain at /Users/me/miniconda3/envs/test_numba_segfault/bin/python (unknown line)
pymain_main at /Users/me/miniconda3/envs/test_numba_segfault/bin/python (unknown line)
main at /Users/me/miniconda3/envs/test_numba_segfault/bin/python (unknown line)
Allocations: 34468216 (Pool: 34457208; Big: 11008); GC: 36
Fatal Python error: PyThreadState_Get: the function must be called with the GIL held, but the GIL is released (the current Python thread state is NULL)
Python runtime state: initialized

Current thread 0x000000011429ae00 (most recent call first):
  File "/Users/me/miniconda3/envs/test_numba_segfault/lib/python3.10/site-packages/llvmlite/binding/ffi.py", line 151 in __call__
  File "/Users/me/miniconda3/envs/test_numba_segfault/lib/python3.10/site-packages/llvmlite/binding/transforms.py", line 88 in _populate_module_pm
  File "/Users/me/miniconda3/envs/test_numba_segfault/lib/python3.10/site-packages/llvmlite/binding/transforms.py", line 95 in populate
  File "/Users/me/miniconda3/envs/test_numba_segfault/lib/python3.10/site-packages/numba/core/codegen.py", line 1212 in _module_pass_manager
  File "/Users/me/miniconda3/envs/test_numba_segfault/lib/python3.10/site-packages/numba/core/codegen.py", line 1191 in _init
  File "/Users/me/miniconda3/envs/test_numba_segfault/lib/python3.10/site-packages/numba/core/codegen.py", line 1172 in __init__
  File "/Users/me/miniconda3/envs/test_numba_segfault/lib/python3.10/site-packages/numba/core/cpu.py", line 50 in init
  File "/Users/me/miniconda3/envs/test_numba_segfault/lib/python3.10/site-packages/numba/core/compiler_lock.py", line 35 in _acquire_compile_lock
  File "/Users/me/miniconda3/envs/test_numba_segfault/lib/python3.10/site-packages/numba/core/base.py", line 262 in __init__
  File "/Users/me/miniconda3/envs/test_numba_segfault/lib/python3.10/site-packages/numba/core/cpu.py", line 41 in __init__
  File "/Users/me/miniconda3/envs/test_numba_segfault/lib/python3.10/site-packages/numba/core/registry.py", line 31 in _toplevel_target_context
  File "/Users/me/miniconda3/envs/test_numba_segfault/lib/python3.10/functools.py", line 981 in __get__
  File "/Users/me/miniconda3/envs/test_numba_segfault/lib/python3.10/site-packages/numba/core/registry.py", line 47 in target_context
  File "/Users/me/miniconda3/envs/test_numba_segfault/lib/python3.10/site-packages/numba/core/dispatcher.py", line 824 in __init__
  File "/Users/me/miniconda3/envs/test_numba_segfault/lib/python3.10/site-packages/numba/core/decorators.py", line 208 in wrapper
  File "/Users/me/miniconda3/envs/test_numba_segfault/lib/python3.10/site-packages/numba/core/decorators.py", line 179 in jit
  File "/Users/me/miniconda3/envs/test_numba_segfault/lib/python3.10/site-packages/numba/core/decorators.py", line 258 in njit
  File "<stdin>", line 1 in <module>

Extension modules: numpy.core._multiarray_umath, numpy.core._multiarray_tests, numpy.linalg.lapack_lite, numpy.linalg._umath_linalg, numpy.fft._pocketfft_internal, numpy.random._common, numpy.random.bit_generator, numpy.random._bounded_integers, numpy.random._mt19937, numpy.random.mtrand, numpy.random._philox, numpy.random._pcg64, numpy.random._sfc64, numpy.random._generator, numba.core.typeconv._typeconv, numba._helperlib, numba._dynfunc, numba._dispatcher, numba.core.runtime._nrt_python, numba.np.ufunc._internal, scipy._lib._ccallback_c (total: 21)

signal (6): Abort trap: 6
in expression starting at none:0
__pthread_kill at /usr/lib/system/libsystem_kernel.dylib (unknown line)
Allocations: 34468216 (Pool: 34457208; Big: 11008); GC: 36
Abort trap: 6

[apmasell]

I have done some research. This is a story but:

• LLVM uses the address of static constants as pass identifiers
• when two incompatible versions of the LLVM are loaded into the same address space, the ID for the two copies of the same pass will not be the same, resulting in the null pointer seen in the trace above
• on Linux and other ELF platforms, two copies can be loaded at once because the symbols are versioned, so the two copies of LLVM will not interfere. MacOS (and the MachO) format, doesn’t have a similar mechanism.
• MachO has a minimum and maximum library versions that should allow limiting the range of library loaded (confusing called current and compatibility versions), but only the lower bound in checked, not the upper bound (verified from the dyld source

Therefore, the only sensible plan to avoid this is to only use static linking on MacOS.

[sklam]

I have a reproducer for folks to experiment on build options to fix this problem. You can find it at https://gist.github.com/sklam/c95e03d54fa1a8ce84cb1247a31b949e