NumPy "dot" hangs when used with multiprocessing (potentially Apple Accelerate related?)

I’m having a devil of a time making a minimal test case, but this seems to be my issue: http://stackoverflow.com/questions/23963997/python-child-process-crashes-on-numpy-dot-if-pyside-is-imported

In my case, I have code that farms out a bunch of calculations, including matrix products to a multiprocessing.Pool with pool.map. The computation hangs partway through, and some hacky print-based debugging shows it hanging on a call to np.dot down in the guts of the program.

Replacing pool.map with the built-in (serial) map makes everything work.

I used not to have this issue, then something changed (lunar eclipse?) and now my computation hangs consistently whenever multiprocessing is used. A minimal test case continues to elude me. (It’s not enough to simply generate 10 random NxN arrays and dot them in a multiprocessing-based way.)

>>> numpy.show_config()
lapack_opt_info:
    extra_link_args = ['-Wl,-framework', '-Wl,Accelerate']
    extra_compile_args = ['-msse3', '-DAPPLE_ACCELERATE_SGEMV_PATCH']
    define_macros = [('NO_ATLAS_INFO', 3)]
openblas_lapack_info:
  NOT AVAILABLE
atlas_3_10_blas_threads_info:
  NOT AVAILABLE
atlas_threads_info:
  NOT AVAILABLE
atlas_3_10_threads_info:
  NOT AVAILABLE
atlas_blas_info:
  NOT AVAILABLE
atlas_3_10_blas_info:
  NOT AVAILABLE
atlas_blas_threads_info:
  NOT AVAILABLE
openblas_info:
  NOT AVAILABLE
blas_mkl_info:
  NOT AVAILABLE
blas_opt_info:
    extra_link_args = ['-Wl,-framework', '-Wl,Accelerate']
    extra_compile_args = ['-msse3', '-DAPPLE_ACCELERATE_SGEMV_PATCH', '-I/System/Library/Frameworks/vecLib.framework/Headers']
    define_macros = [('NO_ATLAS_INFO', 3)]
atlas_info:
  NOT AVAILABLE
atlas_3_10_info:
  NOT AVAILABLE
lapack_mkl_info:
  NOT AVAILABLE
mkl_info:
  NOT AVAILABLE

[ghost]

setting:

os.environ.update(MKL_NUM_THREADS='1')

fixed the issue for me

[ogrisel]

It can be related. For Python 2.7 users (and Python 3 users as well), we started an alternative module to safely work with sub-processes without fearing those kinds of crashes and hanging:

https://github.com/tomMoral/loky

It’s still beta but we would be glad to get your feedback as github issues.