BUG: `integrate/tests/test__quad_vec.py::test_quad_vec_pool` fails with fatal Python error

Describe your issue.

I’m running into an error when running the unit tests in order to validate my installation (note: I’ve build Scipy from sources). However, it only errors out when running the full test suite:

import numpy; import scipy; scipy.test(verbose=2)
...
integrate/tests/test__quad_vec.py::test_quad_vec_pool Fatal Python error: Aborted
...

When running only the particular test file, that same test (and all test from that test file) passes.

import numpy; import scipy; scipy.test(verbose=2, tests=["scipy.integrate.tests.test__quad_vec"])
...
::test_quad_vec_pool PASSED
...

I’m sure this will be a hard one to reproduce. I’m guessing it maybe be due to how the new process is created by multiprocessing, and that somehow this is different between the two different approaches above to run the test.

Note that for me personally, this issue is not a blocker: I can simply assume the installation is ok, considering the 2nd approach of running the tests works. However, I wanted to report it to make you aware, and see if we can somehow resolve this in future versions. In particular, I’m hoping one of the devs can point out what the fundamental differences is between the above two runs, in order to pinpoint the fundamental cause.

Reproducing Code Example

import numpy; import scipy; scipy.test(verbose=2)

Error message

integrate/tests/test__quad_vec.py::test_quad_vec_pool Fatal Python error: Aborted

Current thread 0x000014c1e33ac740 (most recent call first):
  File "/.../Python/3.10.4-GCCcore-11.3.0/lib/python3.10/multiprocessing/popen_fork.py", line 66 in _launch
  File "/.../Python/3.10.4-GCCcore-11.3.0/lib/python3.10/multiprocessing/popen_fork.py", line 19 in __init__
  File "/.../Python/3.10.4-GCCcore-11.3.0/lib/python3.10/multiprocessing/context.py", line 277 in _Popen
  File "/.../Python/3.10.4-GCCcore-11.3.0/lib/python3.10/multiprocessing/process.py", line 121 in start
  File "/.../Python/3.10.4-GCCcore-11.3.0/lib/python3.10/multiprocessing/pool.py", line 326 in _repopulate_pool_static
  File "/.../Python/3.10.4-GCCcore-11.3.0/lib/python3.10/multiprocessing/pool.py", line 303 in _repopulate_pool
  File "/.../Python/3.10.4-GCCcore-11.3.0/lib/python3.10/multiprocessing/pool.py", line 212 in __init__
  File "/.../Python/3.10.4-GCCcore-11.3.0/lib/python3.10/multiprocessing/context.py", line 119 in Pool
  File "/dev/shm/eb-hEvTJj/tmpiAXoaw/lib/python3.10/site-packages/scipy/_lib/_util.py", line 446 in __init__
  File "/dev/shm/eb-hEvTJj/tmpiAXoaw/lib/python3.10/site-packages/scipy/integrate/_quad_vec.py", line 354 in quad_vec
  File "/dev/shm/eb-hEvTJj/tmpiAXoaw/lib/python3.10/site-packages/scipy/integrate/_quad_vec.py", line 264 in quad_vec
  File "/dev/shm/eb-hEvTJj/tmpiAXoaw/lib/python3.10/site-packages/scipy/integrate/tests/test__quad_vec.py", line 108 in test_quad_vec_pool
  File "/.../Python/3.10.4-GCCcore-11.3.0/lib/python3.10/site-packages/_pytest/python.py", line 192 in pytest_pyfunc_call
  File "/.../Python/3.10.4-GCCcore-11.3.0/lib/python3.10/site-packages/pluggy/_callers.py", line 39 in _multicall
  File "/.../Python/3.10.4-GCCcore-11.3.0/lib/python3.10/site-packages/pluggy/_manager.py", line 80 in _hookexec
  File "/.../Python/3.10.4-GCCcore-11.3.0/lib/python3.10/site-packages/pluggy/_hooks.py", line 265 in __call__
  File "/.../Python/3.10.4-GCCcore-11.3.0/lib/python3.10/site-packages/_pytest/python.py", line 1761 in runtest
  File "/.../Python/3.10.4-GCCcore-11.3.0/lib/python3.10/site-packages/_pytest/runner.py", line 166 in pytest_runtest_call
  File "/.../Python/3.10.4-GCCcore-11.3.0/lib/python3.10/site-packages/pluggy/_callers.py", line 39 in _multicall
  File "/.../Python/3.10.4-GCCcore-11.3.0/lib/python3.10/site-packages/pluggy/_manager.py", line 80 in _hookexec
  File "/.../Python/3.10.4-GCCcore-11.3.0/lib/python3.10/site-packages/pluggy/_hooks.py", line 265 in __call__
  File "/.../Python/3.10.4-GCCcore-11.3.0/lib/python3.10/site-packages/_pytest/runner.py", line 259 in <lambda>
  File "/.../Python/3.10.4-GCCcore-11.3.0/lib/python3.10/site-packages/_pytest/runner.py", line 338 in from_call
  File "/.../Python/3.10.4-GCCcore-11.3.0/lib/python3.10/site-packages/_pytest/runner.py", line 258 in call_runtest_hook
  File "/.../Python/3.10.4-GCCcore-11.3.0/lib/python3.10/site-packages/_pytest/runner.py", line 219 in call_and_report
  File "Fatal Python error: /.../Python/3.10.4-GCCcore-11.3.0/lib/python3.10/site-packages/_pytest/runner.py"Aborted, line

130 in runtestprotocol
Current thread 0x  File "000014c1e33ac740/.../Python/3.10.4-GCCcore-11.3.0/lib/python3.10/site-packages/_pytest/runner.py (most recent call first):
", line   File 111" in /.../Python/3.10.4-GCCcore-11.3.0/lib/python3.10/multiprocessing/popen_fork.pypytest_runtest_protocol"
, line   File 66" in /.../Python/3.10.4-GCCcore-11.3.0/lib/python3.10/site-packages/pluggy/_callers.py_launch"
, line 39  File  in "_multicall/.../Python/3.10.4-GCCcore-11.3.0/lib/python3.10/multiprocessing/popen_fork.py
"  File , line "19/.../Python/3.10.4-GCCcore-11.3.0/lib/python3.10/site-packages/pluggy/_manager.py in "__init__, line
80  File  in "_hookexec/.../Python/3.10.4-GCCcore-11.3.0/lib/python3.10/multiprocessing/context.py
"  File , line "277/.../Python/3.10.4-GCCcore-11.3.0/lib/python3.10/site-packages/pluggy/_hooks.py in "_Popen, line
265 in   File __call__"
/.../Python/3.10.4-GCCcore-11.3.0/lib/python3.10/multiprocessing/process.py  File "", line /.../Python/3.10.4-GCCcore-11.3.0/lib/python3.10/site-packages/_pytest/main.py121" in , line
start347
 in pytest_runtestloop  File
"  File /.../Python/3.10.4-GCCcore-11.3.0/lib/python3.10/multiprocessing/pool.py""/.../Python/3.10.4-GCCcore-11.3.0/lib/python3.10/site-packages/pluggy/_callers.py, line "326, line  in
 39_repopulate_pool_static in
_multicall  File
"  File /.../Python/3.10.4-GCCcore-11.3.0/lib/python3.10/multiprocessing/pool.py""/.../Python/3.10.4-GCCcore-11.3.0/lib/python3.10/site-packages/pluggy/_manager.py, line "303, line  in
 80_repopulate_pool in
_hookexec  File
"  File /.../Python/3.10.4-GCCcore-11.3.0/lib/python3.10/multiprocessing/pool.py""/.../Python/3.10.4-GCCcore-11.3.0/lib/python3.10/site-packages/pluggy/_hooks.py, line "212, line  in 2
65__init__ in
__call__
  File   File ""/.../Python/3.10.4-GCCcore-11.3.0/lib/python3.10/multiprocessing/context.py/.../Python/3.10.4-GCCcore-11.3.0/lib/python3.10/site-packages/_pytest/main.py"", line , line
 119322 in  in Pool_main

  File   File ""/dev/shm/eb-hEvTJj/tmpiAXoaw/lib/python3.10/site-packages/scipy/_lib/_util.py/.../Python/3.10.4-GCCcore-11.3.0/lib/python3.10/site-packages/_pytest/main.py"", line , line 446268 in  in __init__wrap_session

  File   File ""/.../Python/3.10.4-GCCcore-11.3.0/lib/python3.10/site-packages/_pytest/main.py/dev/shm/eb-hEvTJj/tmpiAXoaw/lib/python3.10/site-packages/scipy/integrate/_quad_vec.py"", line 315, line  in 354pytest_cmdline_main in
quad_vec  File
"  File /.../Python/3.10.4-GCCcore-11.3.0/lib/python3.10/site-packages/pluggy/_callers.py""/dev/shm/eb-hEvTJj/tmpiAXoaw/lib/python3.10/site-packages/scipy/integrate/_quad_vec.py, line "39, line  in 264_multicall in
quad_vec  File
"/.../Python/3.10.4-GCCcore-11.3.0/lib/python3.10/site-packages/pluggy/_manager.py  File "", line /dev/shm/eb-hEvTJj/tmpiAXoaw/lib/python3.10/site-packages/scipy/integrate/tests/test__quad_vec.py80" in , line _hookexec108
 in   File test_quad_vec_pool"
/.../Python/3.10.4-GCCcore-11.3.0/lib/python3.10/site-packages/pluggy/_hooks.py", line   File 265" in /.../Python/3.10.4-GCCcore-11.3.0/lib/python3.10/site-packages/_pytest/python.py__
call__"
, line 192  File  in "pytest_pyfunc_call/.../Python/3.10.4-GCCcore-11.3.0/lib/python3.10/site-packages/_pytest/config/__init__.py
"  File , line "164/.../Python/3.10.4-GCCcore-11.3.0/lib/python3.10/site-packages/pluggy/_callers.py in "main, line
39 in   File _multicall"
/dev/shm/eb-hEvTJj/tmpiAXoaw/lib/python3.10/site-packages/scipy/_lib/_testutils.py", line   File 69" in /.../Python/3.10.4-GCCcore-11.3.0/lib/python3.10/site-packages/pluggy/_manager.py__call__"
, line 80 in _hookexec  File
"<string>  File "", line /.../Python/3.10.4-GCCcore-11.3.0/lib/python3.10/site-packages/pluggy/_hooks.py1" in , line <module>265
 in __call__
  File "/.../Python/3.10.4-GCCcore-11.3.0/lib/python3.10/site-packages/_pytest/python.py", line 1761 in runtest
  File "/.../Python/3.10.4-GCCcore-11.3.0/lib/python3.10/site-packages/_pytest/runner.py", line 166 in pytest_runtest_call
  File "/.../Python/3.10.4-GCCcore-11.3.0/lib/python3.10/site-packages/pluggy/_callers.py", line 39 in _multicall
  File "/.../Python/3.10.4-GCCcore-11.3.0/lib/python3.10/site-packages/pluggy/_manager.py", line 80 in _hookexec
  File "/.../Python/3.10.4-GCCcore-11.3.0/lib/python3.10/site-packages/pluggy/_hooks.py", line 265 in __call__
  File "/.../Python/3.10.4-GCCcore-11.3.0/lib/python3.10/site-packages/_pytest/runner.py", line 259 in <lambda>
  File "/.../Python/3.10.4-GCCcore-11.3.0/lib/python3.10/site-packages/_pytest/runner.py", line 338 in from_call
  File "/.../Python/3.10.4-GCCcore-11.3.0/lib/python3.10/site-packages/_pytest/runner.py"
Extension modules: , line numpy.core._multiarray_umath258 in call_runtest_hook
  File "/.../Python/3.10.4-GCCcore-11.3.0/lib/python3.10/site-packages/_pytest/runner.py", , line numpy.core._multiarray_tests219 in call_and_report
  File "/.../Python/3.10.4-GCCcore-11.3.0/lib/python3.10/site-packages/_pytest/runner.py", line 130 in runtestprotocol
, numpy.linalg._umath_linalg  File ", /.../Python/3.10.4-GCCcore-11.3.0/lib/python3.10/site-packages/_pytest/runner.pynumpy.fft._pocketfft_internal", line 111,  in numpy.random._commonpytest_runtest_protocol
  File "/.../Python/3.10.4-GCCcore-11.3.0/lib/python3.10/site-packages/pluggy/_callers.py", line 39 in _multicall
  File "/.../Python/3.10.4-GCCcore-11.3.0/lib/python3.10/site-packages/pluggy/_manager.py", line 80 in _hookexec
  File , "numpy.random.bit_generator/.../Python/3.10.4-GCCcore-11.3.0/lib/python3.10/site-packages/pluggy/_hooks.py", line , 265numpy.random._bounded_integers in __call__
, numpy.random._mt19937  File ", /.../Python/3.10.4-GCCcore-11.3.0/lib/python3.10/site-packages/_pytest/main.pynumpy.random.mtrand", line 347 in pytest_runtestloop
,   File numpy.random._philox"/.../Python/3.10.4-GCCcore-11.3.0/lib/python3.10/site-packages/pluggy/_callers.py", line 39,  in numpy.random._pcg64_multicall
,   File numpy.random._sfc64"/.../Python/3.10.4-GCCcore-11.3.0/lib/python3.10/site-packages/pluggy/_manager.py", line , 80numpy.random._generator in _hookexec
  File "/.../Python/3.10.4-GCCcore-11.3.0/lib/python3.10/site-packages/pluggy/_hooks.py", , line scipy._lib._ccallback_c265 in __call__
  File "/.../Python/3.10.4-GCCcore-11.3.0/lib/python3.10/site-packages/_pytest/main.py", line 322 in _main
  File "/.../Python/3.10.4-GCCcore-11.3.0/lib/python3.10/site-packages/_pytest/main.py", line 268 in wrap_session
  File "/.../Python/3.10.4-GCCcore-11.3.0/lib/python3.10/site-packages/_pytest/main.py", line 315 in pytest_cmdline_main
  File "/.../Python/3.10.4-GCCcore-11.3.0/lib/python3.10/site-packages/pluggy/_callers.py", line 39 in _multicall
  File "/.../Python/3.10.4-GCCcore-11.3.0/lib/python3.10/site-packages/pluggy/_manager.py", line 80 in _hookexec
  File "/.../Python/3.10.4-GCCcore-11.3.0/lib/python3.10/site-packages/pluggy/_hooks.py, "numpy.linalg.lapack_lite, line 265 in __call__,
scipy._lib._fpumode  File "/.../Python/3.10.4-GCCcore-11.3.0/lib/python3.10/site-packages/_pytest/config/__init__.py", line 164 in main
  File , "scipy._lib._test_ccallback/dev/shm/eb-hEvTJj/tmpiAXoaw/lib/python3.10/site-packages/scipy/_lib/_testutils.py", , line scipy.sparse._sparsetools69 in __call__
, scipy.sparse._csparsetools  File , "scipy.sparse.csgraph._tools<string>", line , 1scipy.sparse.csgraph._shortest_path in <module>
, scipy.sparse.csgraph._traversal, scipy.sparse.csgraph._min_spanning_tree, scipy.sparse.csgraph._flow, scipy.sparse.csgraph._matching, scipy.sparse.csgraph._reordering, scipy.spatial._ckdtree, scipy._lib.messagestream, scipy.spatial._qhull, scipy.spatial._voronoi, scipy.linalg._fblas, s
cipy.linalg._flapack, scipy.linalg._cythonized_array_utils, scipy.linalg._flinalg, scipy.linalg._solve_toeplitz, scipy.linalg._matfuncs_sqrtm_triu, scipy.linalg.cython_blas, scipy.linalg.cython_lapack, scipy.linalg._decomp_update, scipy.spatial._distance_wrap, scipy.spatial._hausdorff, s
cipy.special._ufuncs_cxx, scipy.special._ufuncs, scipy.special._specfun, scipy.special._comb
Extension modules: , numpy.core._multiarray_umathscipy.special._ellip_harm_2, scipy.spatial.transform._rotation, scipy.cluster._vq, scipy.cluster._hierarchy, scipy.cluster._optimal_leaf_ordering, numpy.core._multiarray_tests, scipy._lib._uarray._uarray, scipy.fftpack.convolve, scipy.spar
se.linalg._isolve._iterative, scipy.sparse.linalg._dsolve._superlu, scipy.sparse.linalg._eigen.arpack._arpack, , scipy.optimize._minpack2numpy.linalg._umath_linalg, scipy.optimize._group_columns, scipy.optimize._trlib._trlib, , numpy.fft._pocketfft_internalscipy.optimize._lbfgsb, , _modu
leTNCnumpy.random._common, scipy.optimize._moduleTNC, scipy.optimize._cobyla, scipy.optimize._slsqp, scipy.optimize._minpack, scipy.optimize._lsq.givens_elimination, scipy.optimize._zeros, numpy.random.bit_generator, scipy.optimize.__nnls, , numpy.random._bounded_integersscipy.optimize._
highs.cython.src._highs_wrapper, , numpy.random._mt19937scipy.optimize._highs._highs_wrapper, , numpy.random.mtrandscipy.optimize._highs.cython.src._highs_constants, , scipy.optimize._highs._highs_constantsnumpy.random._philox, scipy.linalg._interpolative, numpy.random._pcg64, scipy.opti
mize._bglu_dense, , numpy.random._sfc64scipy.optimize._lsap_module, numpy.random._generator, scipy.integrate._odepack, scipy.integrate._quadpack, scipy.integrate._vode, , scipy.integrate._dopscipy._lib._ccallback_c, scipy.integrate._lsoda, scipy.integrate._test_odeint_banded, scipy.interpolate._fitpack, scipy.interpolate.dfitpack, scipy.interpolate._bspl, scipy.interpolate._ppoly, scipy.interpolate.interpnd, scipy.ndimage._nd_image, _ni_label, scipy.ndimage._ni_label, scipy.special.cython_special, scipy.stats._stats, beta_ufunc, scipy.stats._boost.beta_ufunc, binom_ufun
c, scipy.stats._boost.binom_ufunc, nbinom_ufunc, scipy.stats._boost.nbinom_ufunc, hypergeom_ufunc, scipy.stats._boost.hypergeom_ufunc, scipy.stats._biasedurn, scipy.stats._statlib, scipy.stats._mvn, scipy.stats._sobol, scipy.stats._qmc_cy, scipy.stats._unuran.unuran_wrapper, scipy.io.mat
lab._mio_utils, scipy.io.matlab._streams, scipy.io.matlab._mio5_utils, scipy.io._test_fortran, numpy.linalg.lapack_lite, scipy.ndimage._ctest, , scipy._lib._fpumode_cytest, scipy.ndimage._cytest, scipy.odr.__odrpack, scipy.optimize.cython_optimize._zeros, scipy.signal._sigtools, scipy.si
gnal._max_len_seq_inner, scipy.signal._upfirdn_apply, scipy.signal._spline, scipy.signal._sosfilt, scipy.signal._spectral, scipy.signal._peak_finding_utils, scipy.sparse.linalg._propack._spropack, scipy._lib._test_ccallback, scipy.sparse.linalg._propack._dpropack, , scipy.sparse.linalg._
propack._cpropackscipy.sparse._sparsetools, scipy.sparse.linalg._propack._zpropack, scipy.sparse._csparsetools, scipy.special._test_round, scipy.sparse.csgraph._tools, psutil._psutil_linux, scipy.sparse.csgraph._shortest_path, psutil._psutil_posix, scipy.sparse.csgraph._traversal, , scip
y.sparse.csgraph._min_spanning_tree_cffi_backend, scipy.sparse.csgraph._flow, scipy._lib._test_deprecation_def, scipy.sparse.csgraph._matching, scipy._lib._test_deprecation_call (total: , 130scipy.sparse.csgraph._reordering)
, scipy.spatial._ckdtree, scipy._lib.messagestream, scipy.spatial._qhull, scipy.spatial._voronoi, scipy.linalg._fblas, scipy.linalg._flapack, scipy.linalg._cythonized_array_utils, scipy.linalg._flinalg, scipy.linalg._solve_toeplitz, scipy.linalg._matfuncs_sqrtm_triu, scipy.linalg.cython_
blas, scipy.linalg.cython_lapack, scipy.linalg._decomp_update, scipy.spatial._distance_wrap, scipy.spatial._hausdorff, scipy.special._ufuncs_cxx, scipy.special._ufuncs, scipy.special._specfun, scipy.special._comb, scipy.special._ellip_harm_2, scipy.spatial.transform._rotation, scipy.clus
ter._vq, scipy.cluster._hierarchy, scipy.cluster._optimal_leaf_ordering, scipy._lib._uarray._uarray, scipy.fftpack.convolve, scipy.sparse.linalg._isolve._iterative, scipy.sparse.linalg._dsolve._superlu, scipy.sparse.linalg._eigen.arpack._arpack, scipy.optimize._minpack2, scipy.optimize._
group_columns, scipy.optimize._trlib._trlib, scipy.optimize._lbfgsb, _moduleTNC, scipy.optimize._moduleTNC, scipy.optimize._cobyla, scipy.optimize._slsqp, scipy.optimize._minpack, scipy.optimize._lsq.givens_elimination, scipy.optimize._zeros, scipy.optimize.__nnls, scipy.optimize._highs.
cython.src._highs_wrapper, scipy.optimize._highs._highs_wrapper, scipy.optimize._highs.cython.src._highs_constants, scipy.optimize._highs._highs_constants, scipy.linalg._interpolative, scipy.optimize._bglu_dense, scipy.optimize._lsap_module, scipy.integrate._odepack, scipy.integrate._qua
dpack, scipy.integrate._vode, scipy.integrate._dop, scipy.integrate._lsoda, scipy.integrate._test_odeint_banded, scipy.interpolate._fitpack, scipy.interpolate.dfitpack, scipy.interpolate._bspl, scipy.interpolate._ppoly, scipy.interpolate.interpnd, scipy.ndimage._nd_image, _ni_label, scip
y.ndimage._ni_label, scipy.special.cython_special, scipy.stats._stats, beta_ufunc, scipy.stats._boost.beta_ufunc, binom_ufunc, scipy.stats._boost.binom_ufunc, nbinom_ufunc, scipy.stats._boost.nbinom_ufunc, hypergeom_ufunc, scipy.stats._boost.hypergeom_ufunc, scipy.stats._biasedurn, scipy
.stats._statlib, scipy.stats._mvn, scipy.stats._sobol, scipy.stats._qmc_cy, scipy.stats._unuran.unuran_wrapper, scipy.io.matlab._mio_utils, scipy.io.matlab._streams, scipy.io.matlab._mio5_utils, scipy.io._test_fortran, scipy.ndimage._ctest, _cytest, scipy.ndimage._cytest, scipy.odr.__odr
pack, scipy.optimize.cython_optimize._zeros, scipy.signal._sigtools, scipy.signal._max_len_seq_inner, scipy.signal._upfirdn_apply, scipy.signal._spline, scipy.signal._sosfilt, scipy.signal._spectral, scipy.signal._peak_finding_utils, scipy.sparse.linalg._propack._spropack, scipy.sparse.l
inalg._propack._dpropack, scipy.sparse.linalg._propack._cpropack, scipy.sparse.linalg._propack._zpropack, scipy.special._test_round, psutil._psutil_linux, psutil._psutil_posix, _cffi_backend, scipy._lib._test_deprecation_def, scipy._lib._test_deprecation_call (total: 130)
Aborted (core dumped)

SciPy/NumPy/Python version information

1.8.1 1.22.3 sys.version_info(major=3, minor=10, micro=4, releaselevel=‘final’, serial=0)

[casparvl]

@tupui I’m building 1.8.1, so not main. i.e. I’m using the 1.8.1 tagged version. Regarding the OS: it’s Rocky Linux 8.6. But note that I’m not using any of the OS libraries to build against. I’m using EasyBuild, a tool that builds software on HPC systems. EasyBuild installs pretty much everything from the ground up, including low level libraries, in order to level the playing field between OS-es. The only thing still used from the OS would be GLIBC.

@rgommers I’m running with GCC 11.3 stack, also build from sources (also with EasyBuild).

Looking at the stack trace, and seeing things like ‘popen_fork.py’ in there, my intuition says it’s a process-creation issues. I wouldn’t think its related to the GCC stack. If it is about process creation, it could be related to the OS of course, but that’s a variable I don’t control (this is a managed HPC system, I’m just a user running in userspace).

What I dont understand is how it then works if I run with

import numpy; import scipy; scipy.test(verbose=2, tests=["scipy.integrate.tests.test__quad_vec"])

Why would process creation somehow be different if the test is launched in this way (compared to not specifying the explicit test and just running the full suite)? If we could understand that, and make a regular scipy.test(verbose=2) use the same type of process launching, maybe we can avoid this problem in the future 😃

[tupui]

Just a note about os.sched_getaffinity which is only available on certain platform, so we would still need to have a fallback strategy.